[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:51:56 dist-test-slave-2dg3 krb5kdc[9197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:51:56 dist-test-slave-2dg3 krb5kdc[9197](info): set up 2 sockets
Sep 26 20:51:56 dist-test-slave-2dg3 krb5kdc[9197](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 20:51:57.464223 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:51:57 dist-test-slave-2dg3 krb5kdc[9197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:51:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:51:57.638754 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:37653
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:37653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:51:57.956822 9210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:51:57.957103 9210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:51:57.957190 9210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:51:57.964344 9210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:51:57.964445 9210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:51:57.964489 9210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:51:57.964545 9210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:51:57.964581 9210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:51:57.974445 9210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:37653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:37653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:51:57.975809 9210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:51:57.977049 9210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:51:57.989303 9215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:51:57.989230 9216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:51:57.989266 9218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:51:57.990482 9210 server_base.cc:1034] running on GCE node
I20260926 20:51:57.991379 9210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:51:57.992841 9210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:51:57.994055 9210 hybrid_clock.cc:648] HybridClock initialized: now 1790455917994002 us; error 45 us; skew 500 ppm
I20260926 20:51:57.994451 9210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:51:57 dist-test-slave-2dg3 krb5kdc[9197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455917, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:51:57.997773 9210 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:51:57.999315 9210 webserver.cc:460] Webserver started at http://127.8.249.190:38933/ using document root <none> and password file <none>
I20260926 20:51:58.000025 9210 fs_manager.cc:359] Metadata directory not provided
I20260926 20:51:58.000157 9210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:51:58.000471 9210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:51:58.003526 9210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "0799915822534965a55c238013027464"
format_stamp: "Formatted at 2026-09-26 20:51:57 on dist-test-slave-2dg3"
server_key: "2bb82e3e60da6ba730278fdd40a3eed7"
server_key_iv: "546d5bae8a745541c2a44da8e641a2f7"
server_key_version: "encryptionkey@0"
I20260926 20:51:58.004359 9210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "0799915822534965a55c238013027464"
format_stamp: "Formatted at 2026-09-26 20:51:57 on dist-test-slave-2dg3"
server_key: "2bb82e3e60da6ba730278fdd40a3eed7"
server_key_iv: "546d5bae8a745541c2a44da8e641a2f7"
server_key_version: "encryptionkey@0"
I20260926 20:51:58.009821 9210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:51:58.013247 9225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:51:58.014585 9210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:51:58.014829 9210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "0799915822534965a55c238013027464"
format_stamp: "Formatted at 2026-09-26 20:51:57 on dist-test-slave-2dg3"
server_key: "2bb82e3e60da6ba730278fdd40a3eed7"
server_key_iv: "546d5bae8a745541c2a44da8e641a2f7"
server_key_version: "encryptionkey@0"
I20260926 20:51:58.015045 9210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:51:58.039330 9210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:51:58.043815 9210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:51:58.044103 9210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:51:58.062362 9210 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:37653
I20260926 20:51:58.062382 9276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:37653 every 8 connection(s)
I20260926 20:51:58.063860 9210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:51:58.066403 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 9210
I20260926 20:51:58.066752 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:51:58.068737 9277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:51:58.069365 9190 external_mini_cluster.cc:1428] Setting key 019204144af0418d1a0da5f76a89c4fd
I20260926 20:51:58.080885 9277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464: Bootstrap starting.
I20260926 20:51:58.084653 9277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464: Neither blocks nor log segments found. Creating new log.
I20260926 20:51:58.086146 9277 log.cc:824] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464: Log is configured to *not* fsync() on all Append() calls
I20260926 20:51:58.089540 9277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464: No bootstrap required, opened a new log
I20260926 20:51:58.098778 9277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0799915822534965a55c238013027464" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 37653 } }
I20260926 20:51:58.099231 9277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:51:58.099357 9277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0799915822534965a55c238013027464, State: Initialized, Role: FOLLOWER
Sep 26 20:51:58 dist-test-slave-2dg3 krb5kdc[9197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:51:58.100032 9277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0799915822534965a55c238013027464" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 37653 } }
I20260926 20:51:58.100299 9277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:51:58.100450 9277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:51:58.100642 9277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:51:58.104350 9277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0799915822534965a55c238013027464" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 37653 } }
I20260926 20:51:58.105014 9277 leader_election.cc:302] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0799915822534965a55c238013027464; no voters:
I20260926 20:51:58.105763 9277 leader_election.cc:288] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:51:58.105942 9282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:51:58.107847 9282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [term 1 LEADER]: Becoming Leader. State: Replica: 0799915822534965a55c238013027464, State: Running, Role: LEADER
I20260926 20:51:58.108067 9277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:51:58.108445 9282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0799915822534965a55c238013027464" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 37653 } }
I20260926 20:51:58.113489 9279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:51:58.074055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.8.249.190:37653)
0926 20:51:58.074990 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0926 20:51:58.075013 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 20:51:58.076219 (+ 1206us) server_negotiation.cc:407] Connection header received
0926 20:51:58.085707 (+ 9488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:51:58.085739 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:51:58.086317 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:51:58.086871 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:51:58.090334 (+ 3463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:58.092472 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:58.094131 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:58.095366 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:58.095572 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:51:58.101145 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:51:58.101158 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:51:58.101182 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:51:58.101238 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:51:58.105438 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:58.106566 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:58.106580 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:58.106592 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:58.106893 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:58.107617 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:58.107623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:58.107627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:58.108485 (+ 858us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:51:58.108825 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 20:51:58.111200 (+ 2375us) server_negotiation.cc:299] Negotiation successful
0926 20:51:58.111729 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":384,"threads_started":1}
I20260926 20:51:58.116312 9284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0799915822534965a55c238013027464. Latest consensus state: current_term: 1 leader_uuid: "0799915822534965a55c238013027464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0799915822534965a55c238013027464" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 37653 } } }
I20260926 20:51:58.116703 9284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [sys.catalog]: This master's current role is: LEADER
I20260926 20:51:58.118100 9283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0799915822534965a55c238013027464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0799915822534965a55c238013027464" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 37653 } } }
I20260926 20:51:58.118965 9283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464 [sys.catalog]: This master's current role is: LEADER
I20260926 20:51:58.120342 9291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:51:58.125074 9291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:51:58.133144 9291 catalog_manager.cc:1347] Generated new cluster ID: 80a2e18a2f4748aa9383190f34e125c1
I20260926 20:51:58.133241 9291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:51:58.156450 9291 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:51:58.157850 9291 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:51:58.174046 9291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0799915822534965a55c238013027464: Generated new TSK 0
I20260926 20:51:58.175204 9291 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:51:58.220537 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:37653
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:51:58.557930 9303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:51:58.558213 9303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:51:58.558365 9303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:51:58.565667 9303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:51:58.565764 9303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:51:58.565924 9303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:51:58.575354 9303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:37653
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:51:58.576613 9303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:51:58.577875 9303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:51:58.591460 9308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:51:58.591508 9309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:51:58.591423 9311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:51:58.592430 9303 server_base.cc:1034] running on GCE node
I20260926 20:51:58.593083 9303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:51:58.594101 9303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:51:58.595323 9303 hybrid_clock.cc:648] HybridClock initialized: now 1790455918595284 us; error 42 us; skew 500 ppm
I20260926 20:51:58.595654 9303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:51:58 dist-test-slave-2dg3 krb5kdc[9197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455918, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:51:58.598670 9303 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:51:58.600283 9303 webserver.cc:460] Webserver started at http://127.8.249.129:35455/ using document root <none> and password file <none>
I20260926 20:51:58.600957 9303 fs_manager.cc:359] Metadata directory not provided
I20260926 20:51:58.601114 9303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:51:58.601420 9303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:51:58.604321 9303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "08f84f27814b4fecb65c377112404ce2"
format_stamp: "Formatted at 2026-09-26 20:51:58 on dist-test-slave-2dg3"
server_key: "07b1b4f0b4a5f4b8612817933bcda9b3"
server_key_iv: "d1c880ff30b3d65835b45fcf888d45a6"
server_key_version: "encryptionkey@0"
I20260926 20:51:58.605067 9303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "08f84f27814b4fecb65c377112404ce2"
format_stamp: "Formatted at 2026-09-26 20:51:58 on dist-test-slave-2dg3"
server_key: "07b1b4f0b4a5f4b8612817933bcda9b3"
server_key_iv: "d1c880ff30b3d65835b45fcf888d45a6"
server_key_version: "encryptionkey@0"
I20260926 20:51:58.610121 9303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 20:51:58.613408 9318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:51:58.614880 9303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:51:58.615051 9303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "08f84f27814b4fecb65c377112404ce2"
format_stamp: "Formatted at 2026-09-26 20:51:58 on dist-test-slave-2dg3"
server_key: "07b1b4f0b4a5f4b8612817933bcda9b3"
server_key_iv: "d1c880ff30b3d65835b45fcf888d45a6"
server_key_version: "encryptionkey@0"
I20260926 20:51:58.615244 9303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:51:58.631369 9303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:51:58.635725 9303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:51:58.636024 9303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:51:58.636993 9303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:51:58.638329 9303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:51:58.638437 9303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:51:58.638589 9303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:51:58.638670 9303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:51:58.671438 9303 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:34313
I20260926 20:51:58.671487 9430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:34313 every 8 connection(s)
I20260926 20:51:58.672844 9303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:51:58.677579 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 9303
I20260926 20:51:58.677806 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:51:58.678402 9190 external_mini_cluster.cc:1428] Setting key 2d9b9eda9e8fde924b023db911e78399
Sep 26 20:51:58 dist-test-slave-2dg3 krb5kdc[9197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455918, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:51:58.694268 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:51:58.675516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:45077 (local address 127.8.249.190:37653)
0926 20:51:58.675923 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 20:51:58.675928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:51:58.676977 (+ 1049us) server_negotiation.cc:407] Connection header received
0926 20:51:58.678969 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:51:58.678974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:51:58.679076 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:51:58.679196 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:51:58.680934 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:58.681639 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:58.683281 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:58.683688 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:58.683835 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:51:58.687349 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:51:58.687358 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:51:58.687367 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:51:58.687414 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:51:58.690954 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:58.691748 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:58.691757 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:58.691763 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:58.691919 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:58.692337 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:58.692345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:58.692352 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:58.692778 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:51:58.692956 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 20:51:58.693615 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 20:51:58.693872 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":213,"threads_started":1}
I20260926 20:51:58.695034 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:51:58.676117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:37653 (local address 127.8.249.129:45077)
0926 20:51:58.676818 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 20:51:58.676861 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:51:58.678657 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:51:58.679471 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:51:58.679484 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:51:58.680220 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:51:58.680775 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:51:58.680790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:51:58.681804 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:51:58.681810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:51:58.683136 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:51:58.683148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:51:58.683920 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:51:58.683928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:51:58.684167 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:51:58.684870 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:51:58.684893 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:51:58.687111 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:51:58.691112 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:51:58.691119 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:51:58.691132 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:51:58.691557 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:51:58.692033 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:51:58.692037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:51:58.692039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:51:58.692214 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:51:58.692905 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:51:58.692910 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:51:58.693231 (+ 321us) client_negotiation.cc:769] Sending connection context
0926 20:51:58.693455 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 20:51:58.693857 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":261,"threads_started":1}
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
I20260926 20:51:58.696683 9431 heartbeater.cc:347] Connected to a master server at 127.8.249.190:37653
I20260926 20:51:58.697023 9431 heartbeater.cc:464] Registering TS with master...
I20260926 20:51:58.697777 9431 heartbeater.cc:511] Master 127.8.249.190:37653 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
I20260926 20:51:58.700127 9242 ts_manager.cc:194] Registered new tserver with Master: 08f84f27814b4fecb65c377112404ce2 (127.8.249.129:34313)
I20260926 20:51:58.702255 9242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:45077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:51:58.721709 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:37653
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:51:59.036449 9437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:51:59.036772 9437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:51:59.036949 9437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:51:59.044169 9437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:51:59.044341 9437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:51:59.044519 9437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:51:59.054366 9437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:37653
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:51:59.055749 9437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:51:59.057159 9437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:51:59.070221 9443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:51:59.070254 9442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:51:59.071074 9445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:51:59.072280 9437 server_base.cc:1034] running on GCE node
I20260926 20:51:59.072984 9437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:51:59.074273 9437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:51:59.075488 9437 hybrid_clock.cc:648] HybridClock initialized: now 1790455919075430 us; error 57 us; skew 500 ppm
I20260926 20:51:59.075825 9437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:51:59 dist-test-slave-2dg3 krb5kdc[9197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455919, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:51:59.079185 9437 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:51:59.080623 9437 webserver.cc:460] Webserver started at http://127.8.249.130:38923/ using document root <none> and password file <none>
I20260926 20:51:59.081290 9437 fs_manager.cc:359] Metadata directory not provided
I20260926 20:51:59.081388 9437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:51:59.081641 9437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:51:59.084534 9437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "7319179b7c6349dc9265eebb62ec413a"
format_stamp: "Formatted at 2026-09-26 20:51:59 on dist-test-slave-2dg3"
server_key: "594765777960e3d49ab46e2120577af1"
server_key_iv: "0a6e1fbaca65125efd1fc9e7e1e600fc"
server_key_version: "encryptionkey@0"
I20260926 20:51:59.085263 9437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "7319179b7c6349dc9265eebb62ec413a"
format_stamp: "Formatted at 2026-09-26 20:51:59 on dist-test-slave-2dg3"
server_key: "594765777960e3d49ab46e2120577af1"
server_key_iv: "0a6e1fbaca65125efd1fc9e7e1e600fc"
server_key_version: "encryptionkey@0"
I20260926 20:51:59.090545 9437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:51:59.094254 9452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:51:59.095745 9437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:51:59.095894 9437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "7319179b7c6349dc9265eebb62ec413a"
format_stamp: "Formatted at 2026-09-26 20:51:59 on dist-test-slave-2dg3"
server_key: "594765777960e3d49ab46e2120577af1"
server_key_iv: "0a6e1fbaca65125efd1fc9e7e1e600fc"
server_key_version: "encryptionkey@0"
I20260926 20:51:59.096067 9437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:51:59.118356 9437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:51:59.122629 9437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:51:59.122949 9437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:51:59.123889 9437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:51:59.125180 9437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:51:59.125254 9437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:51:59.125344 9437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:51:59.125387 9437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:51:59.158382 9437 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:44051
I20260926 20:51:59.158427 9564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:44051 every 8 connection(s)
I20260926 20:51:59.159809 9437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:51:59.168565 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 9437
I20260926 20:51:59.168787 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:51:59.169345 9190 external_mini_cluster.cc:1428] Setting key 736d4f5d534ac9feb09e440b0a7d50db
Sep 26 20:51:59 dist-test-slave-2dg3 krb5kdc[9197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455919, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:51:59.181433 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:51:59.162449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:49707 (local address 127.8.249.190:37653)
0926 20:51:59.162658 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 20:51:59.162665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:51:59.163930 (+ 1265us) server_negotiation.cc:407] Connection header received
0926 20:51:59.165767 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:51:59.165771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:51:59.165837 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:51:59.165931 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:51:59.167583 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.168335 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.170572 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.171150 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.171293 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:51:59.174866 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:51:59.174876 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:51:59.174884 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:51:59.174937 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:51:59.177784 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:59.178861 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:59.178870 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:59.178872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:59.178999 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:59.179527 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:59.179532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:59.179534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:59.179872 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:51:59.179995 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:51:59.180933 (+ 938us) server_negotiation.cc:299] Negotiation successful
0926 20:51:59.181139 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":72}
I20260926 20:51:59.183065 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:51:59.162991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:37653 (local address 127.8.249.130:49707)
0926 20:51:59.163752 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 20:51:59.163804 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:51:59.165503 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:51:59.166157 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:51:59.166166 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:51:59.166736 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:51:59.167424 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:51:59.167443 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.168523 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.168529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:51:59.170365 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:51:59.170379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.171400 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.171407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:51:59.171680 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:51:59.172411 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:51:59.172434 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:51:59.174599 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:51:59.177945 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:51:59.177958 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:51:59.177980 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:51:59.178686 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:51:59.179109 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:51:59.179117 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:51:59.179123 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:51:59.179384 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:51:59.180030 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:51:59.180041 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:51:59.180523 (+ 482us) client_negotiation.cc:769] Sending connection context
0926 20:51:59.180839 (+ 316us) client_negotiation.cc:240] Negotiation successful
0926 20:51:59.181443 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":296,"threads_started":1}
I20260926 20:51:59.184772 9565 heartbeater.cc:347] Connected to a master server at 127.8.249.190:37653
I20260926 20:51:59.185106 9565 heartbeater.cc:464] Registering TS with master...
I20260926 20:51:59.185882 9565 heartbeater.cc:511] Master 127.8.249.190:37653 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
I20260926 20:51:59.187445 9242 ts_manager.cc:194] Registered new tserver with Master: 7319179b7c6349dc9265eebb62ec413a (127.8.249.130:44051)
I20260926 20:51:59.188285 9242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:49707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:51:59.211926 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:37653
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:51:59.529611 9570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:51:59.529950 9570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:51:59.530128 9570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:51:59.537407 9570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:51:59.537590 9570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:51:59.537781 9570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:51:59.548004 9570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:37653
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:51:59.549355 9570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:51:59.550690 9570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:51:59.563711 9575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:51:59.563799 9576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:51:59.563757 9578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:51:59.564695 9570 server_base.cc:1034] running on GCE node
I20260926 20:51:59.565260 9570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:51:59.566164 9570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:51:59.567401 9570 hybrid_clock.cc:648] HybridClock initialized: now 1790455919567343 us; error 40 us; skew 500 ppm
I20260926 20:51:59.567718 9570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:51:59 dist-test-slave-2dg3 krb5kdc[9197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455919, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:51:59.570988 9570 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:51:59.572656 9570 webserver.cc:460] Webserver started at http://127.8.249.131:44043/ using document root <none> and password file <none>
I20260926 20:51:59.573385 9570 fs_manager.cc:359] Metadata directory not provided
I20260926 20:51:59.573493 9570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:51:59.573761 9570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:51:59.576769 9570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "51bd74768b174675bebb5054b3a91ceb"
format_stamp: "Formatted at 2026-09-26 20:51:59 on dist-test-slave-2dg3"
server_key: "21d6e1de2789ce6e2e6169c240c60ed2"
server_key_iv: "5a34e18751c3d65b4ecca516f8dfad3a"
server_key_version: "encryptionkey@0"
I20260926 20:51:59.577499 9570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "51bd74768b174675bebb5054b3a91ceb"
format_stamp: "Formatted at 2026-09-26 20:51:59 on dist-test-slave-2dg3"
server_key: "21d6e1de2789ce6e2e6169c240c60ed2"
server_key_iv: "5a34e18751c3d65b4ecca516f8dfad3a"
server_key_version: "encryptionkey@0"
I20260926 20:51:59.582641 9570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:51:59.586011 9585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:51:59.587517 9570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:51:59.587656 9570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "51bd74768b174675bebb5054b3a91ceb"
format_stamp: "Formatted at 2026-09-26 20:51:59 on dist-test-slave-2dg3"
server_key: "21d6e1de2789ce6e2e6169c240c60ed2"
server_key_iv: "5a34e18751c3d65b4ecca516f8dfad3a"
server_key_version: "encryptionkey@0"
I20260926 20:51:59.587819 9570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:51:59.606710 9570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:51:59.611569 9570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:51:59.611876 9570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:51:59.612841 9570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:51:59.614192 9570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:51:59.614262 9570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:51:59.614427 9570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:51:59.614477 9570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:51:59.646523 9570 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:43115
I20260926 20:51:59.646559 9697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:43115 every 8 connection(s)
I20260926 20:51:59.647863 9570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:51:59.650147 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 9570
I20260926 20:51:59.650352 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:51:59.650852 9190 external_mini_cluster.cc:1428] Setting key 0bfccbf40da3e444044b43e86aec24f8
Sep 26 20:51:59 dist-test-slave-2dg3 krb5kdc[9197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455919, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:51:59.668807 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:51:59.650435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:55373 (local address 127.8.249.190:37653)
0926 20:51:59.650619 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:51:59.650625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:51:59.652116 (+ 1491us) server_negotiation.cc:407] Connection header received
0926 20:51:59.654428 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:51:59.654433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:51:59.654518 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:51:59.654636 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:51:59.656244 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.656994 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.658568 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.659050 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.659209 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:51:59.662711 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:51:59.662716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:51:59.662720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:51:59.662768 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:51:59.665635 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:59.666613 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:59.666617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:59.666619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:59.666832 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:59.667350 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:59.667353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:59.667355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:59.667643 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:51:59.667752 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:51:59.668409 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 20:51:59.668568 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:51:59.669840 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:51:59.651070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:37653 (local address 127.8.249.131:55373)
0926 20:51:59.651941 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0926 20:51:59.651993 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:51:59.654141 (+ 2148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:51:59.654933 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:51:59.654942 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:51:59.655465 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:51:59.656049 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:51:59.656066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.657160 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.657166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:51:59.658409 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:51:59.658421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.659320 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.659327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:51:59.659566 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:51:59.660210 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:51:59.660232 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:51:59.662483 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:51:59.665789 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:51:59.665798 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:51:59.665816 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:51:59.666473 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:51:59.666943 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:51:59.666946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:51:59.666948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:51:59.667205 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:51:59.667760 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:51:59.667766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:51:59.668105 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 20:51:59.668349 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 20:51:59.668735 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":303,"threads_started":1}
I20260926 20:51:59.671285 9698 heartbeater.cc:347] Connected to a master server at 127.8.249.190:37653
I20260926 20:51:59.671597 9698 heartbeater.cc:464] Registering TS with master...
I20260926 20:51:59.672312 9698 heartbeater.cc:511] Master 127.8.249.190:37653 requested a full tablet report, sending...
I20260926 20:51:59.673885 9242 ts_manager.cc:194] Registered new tserver with Master: 51bd74768b174675bebb5054b3a91ceb (127.8.249.131:43115)
I20260926 20:51:59.675036 9242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:55373
I20260926 20:51:59.679399 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:51:59 dist-test-slave-2dg3 krb5kdc[9197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455919, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 20:51:59.705295 9431 heartbeater.cc:503] Master 127.8.249.190:37653 was elected leader, sending a full tablet report...
Sep 26 20:51:59 dist-test-slave-2dg3 krb5kdc[9197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455919, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:51:59.726620 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:51:59.710399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.8.249.129:34313)
0926 20:51:59.711112 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0926 20:51:59.711122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:51:59.711231 (+ 109us) server_negotiation.cc:407] Connection header received
0926 20:51:59.711458 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:51:59.711465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:51:59.711723 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:51:59.711904 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:51:59.713402 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.714385 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.715730 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.716388 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.716529 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:51:59.720999 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:51:59.721012 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:51:59.721027 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:51:59.721080 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:51:59.724014 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:59.724759 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:59.724766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:59.724768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:59.724890 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:59.725310 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:59.725313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:59.725315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:59.725651 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:51:59.725865 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 20:51:59.726132 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 20:51:59.726338 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":319,"threads_started":1}
W20260926 20:51:59.727943 9346 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51718
Sep 26 20:51:59 dist-test-slave-2dg3 krb5kdc[9197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:51:59 dist-test-slave-2dg3 krb5kdc[9197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:51:59.771296 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:51:59.757900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.8.249.129:34313)
0926 20:51:59.758130 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 20:51:59.758137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:51:59.758408 (+ 271us) server_negotiation.cc:407] Connection header received
0926 20:51:59.758618 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:51:59.758622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:51:59.758692 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:51:59.758806 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:51:59.760125 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.760796 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.761992 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.762432 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.762562 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:51:59.766355 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:51:59.766360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:51:59.766363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:51:59.766399 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:51:59.768795 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:59.769440 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:59.769449 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:59.769452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:59.769563 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:59.769918 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:59.769925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:59.769928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:59.770174 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:51:59.770419 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 20:51:59.770707 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 20:51:59.770963 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 20:51:59.772719 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 9303
I20260926 20:51:59.784662 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:51:59.773650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.8.249.129:34313)
0926 20:51:59.773828 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:51:59.773834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:51:59.774195 (+ 361us) server_negotiation.cc:407] Connection header received
0926 20:51:59.774311 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:51:59.774314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:51:59.774373 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:51:59.774448 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:51:59.775607 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.776272 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.777448 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.777838 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.777924 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:51:59.780159 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:51:59.780163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:51:59.780166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:51:59.780194 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:51:59.782472 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:59.783113 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:59.783117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:59.783120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:59.783242 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:59.783547 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:59.783550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:59.783552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:59.783782 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:51:59.783947 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 20:51:59.784197 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 20:51:59.784356 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 20:51:59.785368 9426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:51:59.868630 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 9303
I20260926 20:51:59.884632 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 9437
Sep 26 20:51:59 dist-test-slave-2dg3 krb5kdc[9197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:51:59.900859 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:51:59.885567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.8.249.130:44051)
0926 20:51:59.886170 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 20:51:59.886178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:51:59.886205 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:51:59.886267 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:51:59.886273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:51:59.886481 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:51:59.886613 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:51:59.887797 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.888789 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.889945 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:51:59.890571 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:51:59.890671 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:51:59.894733 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:51:59.894746 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:51:59.894761 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:51:59.894870 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:51:59.897717 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:59.898294 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:59.898301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:59.898304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:59.898473 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:51:59.899018 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:51:59.899027 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:51:59.899033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:51:59.899614 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:51:59.899866 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 20:51:59.900155 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 20:51:59.900440 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":330,"threads_started":1}
I20260926 20:51:59.902166 9560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:51:59.984433 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 9437
I20260926 20:52:00.000886 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 9570
Sep 26 20:52:00 dist-test-slave-2dg3 krb5kdc[9197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:52:00.017684 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:00.001989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.8.249.131:43115)
0926 20:52:00.002563 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:00.002570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:00.002599 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:52:00.002656 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:00.002662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:00.002940 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:00.003066 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:00.004347 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:00.005342 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:00.006621 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:00.007336 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:00.007427 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:00.011980 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:00.011992 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:00.012008 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:00.012044 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:00.014771 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:00.015545 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:00.015561 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:00.015563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:00.015679 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:00.016086 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:00.016092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:00.016095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:00.016436 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:00.016750 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:00.017085 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 20:52:00.017359 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":267,"threads_started":1}
I20260926 20:52:00.019101 9693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:00.106591 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 9570
I20260926 20:52:00.122211 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 9210
I20260926 20:52:00.123464 9272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:00.225078 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 9210
2026-09-26T20:52:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3860 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:52:00 dist-test-slave-2dg3 krb5kdc[9719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:52:00 dist-test-slave-2dg3 krb5kdc[9719](info): set up 2 sockets
Sep 26 20:52:00 dist-test-slave-2dg3 krb5kdc[9719](info): commencing operation
krb5kdc: starting...
W20260926 20:52:01.298961 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:52:01 dist-test-slave-2dg3 krb5kdc[9719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:52:01.486999 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:46535
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:46535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:01.810500 9732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:01.810815 9732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:01.810956 9732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:01.818183 9732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:01.818303 9732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:01.818349 9732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:01.818410 9732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:01.818454 9732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:01.828660 9732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:46535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:46535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:01.830018 9732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:01.831477 9732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:01.843845 9737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:01.843832 9738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:01.843885 9740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:01.844703 9732 server_base.cc:1034] running on GCE node
I20260926 20:52:01.845510 9732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:01.847090 9732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:01.848322 9732 hybrid_clock.cc:648] HybridClock initialized: now 1790455921848274 us; error 56 us; skew 500 ppm
I20260926 20:52:01.848670 9732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:01 dist-test-slave-2dg3 krb5kdc[9719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455921, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:01.854964 9732 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:52:01.856714 9732 webserver.cc:460] Webserver started at http://127.8.249.190:42913/ using document root <none> and password file <none>
I20260926 20:52:01.857484 9732 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:01.857635 9732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:01.857955 9732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:01.861092 9732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "d6d7ac8305cf49bc8ec3d0db254a53a9"
format_stamp: "Formatted at 2026-09-26 20:52:01 on dist-test-slave-2dg3"
server_key: "608f59ff2ee1023055b1051ca0dd7aa6"
server_key_iv: "b95b94d5af882b8bf7d6a98f4b450add"
server_key_version: "encryptionkey@0"
I20260926 20:52:01.861888 9732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "d6d7ac8305cf49bc8ec3d0db254a53a9"
format_stamp: "Formatted at 2026-09-26 20:52:01 on dist-test-slave-2dg3"
server_key: "608f59ff2ee1023055b1051ca0dd7aa6"
server_key_iv: "b95b94d5af882b8bf7d6a98f4b450add"
server_key_version: "encryptionkey@0"
I20260926 20:52:01.867568 9732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:52:01.871351 9747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:01.872783 9732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:52:01.872978 9732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "d6d7ac8305cf49bc8ec3d0db254a53a9"
format_stamp: "Formatted at 2026-09-26 20:52:01 on dist-test-slave-2dg3"
server_key: "608f59ff2ee1023055b1051ca0dd7aa6"
server_key_iv: "b95b94d5af882b8bf7d6a98f4b450add"
server_key_version: "encryptionkey@0"
I20260926 20:52:01.873198 9732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:01.903033 9732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:01.907629 9732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:01.907991 9732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:01.928464 9798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:46535 every 8 connection(s)
I20260926 20:52:01.928494 9732 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:46535
I20260926 20:52:01.930691 9732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:01.935060 9799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:01.935051 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 9732
I20260926 20:52:01.935287 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:01.936101 9190 external_mini_cluster.cc:1428] Setting key 4aa573d504cb281a7f9b2f368af7508c
I20260926 20:52:01.950171 9799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9: Bootstrap starting.
Sep 26 20:52:01 dist-test-slave-2dg3 krb5kdc[9719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:01.953820 9799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:01.954955 9799 log.cc:824] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:01.957998 9799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9: No bootstrap required, opened a new log
I20260926 20:52:01.960712 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:01.938693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33502 (local address 127.8.249.190:46535)
0926 20:52:01.939762 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:01.939775 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:01.939832 (+ 57us) server_negotiation.cc:407] Connection header received
0926 20:52:01.941956 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:01.941987 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:01.942452 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:01.942922 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:01.944658 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:01.945993 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:01.947851 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:01.948543 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:01.948676 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:01.953317 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:01.953326 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:01.953342 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:01.953387 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:01.956584 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:01.957282 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:01.957289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:01.957296 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:01.957458 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:01.957836 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:01.957840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:01.957842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:01.958403 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:01.958696 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:01.959254 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 20:52:01.959660 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":523,"threads_started":1}
I20260926 20:52:01.966203 9799 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d7ac8305cf49bc8ec3d0db254a53a9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46535 } }
I20260926 20:52:01.966702 9799 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:01.966852 9799 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d7ac8305cf49bc8ec3d0db254a53a9, State: Initialized, Role: FOLLOWER
I20260926 20:52:01.967446 9799 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d7ac8305cf49bc8ec3d0db254a53a9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46535 } }
I20260926 20:52:01.967672 9799 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:01.967800 9799 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:01.967968 9799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:01.971674 9799 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d7ac8305cf49bc8ec3d0db254a53a9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46535 } }
I20260926 20:52:01.972218 9799 leader_election.cc:302] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6d7ac8305cf49bc8ec3d0db254a53a9; no voters:
I20260926 20:52:01.972748 9799 leader_election.cc:288] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:01.972885 9804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:01.973950 9804 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [term 1 LEADER]: Becoming Leader. State: Replica: d6d7ac8305cf49bc8ec3d0db254a53a9, State: Running, Role: LEADER
I20260926 20:52:01.974520 9804 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d7ac8305cf49bc8ec3d0db254a53a9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46535 } }
I20260926 20:52:01.974818 9799 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:01.977413 9806 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6d7ac8305cf49bc8ec3d0db254a53a9. Latest consensus state: current_term: 1 leader_uuid: "d6d7ac8305cf49bc8ec3d0db254a53a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d7ac8305cf49bc8ec3d0db254a53a9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46535 } } }
I20260926 20:52:01.977856 9806 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:01.977403 9805 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6d7ac8305cf49bc8ec3d0db254a53a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d7ac8305cf49bc8ec3d0db254a53a9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46535 } } }
I20260926 20:52:01.978264 9805 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:01.981101 9810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:01.985431 9810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:01.994433 9810 catalog_manager.cc:1347] Generated new cluster ID: 0dc9be06719a418bb8ac4fdf68f4bb3a
I20260926 20:52:01.994560 9810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:02.014410 9810 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:02.015906 9810 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:02.027238 9810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6d7ac8305cf49bc8ec3d0db254a53a9: Generated new TSK 0
I20260926 20:52:02.028460 9810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:52:02.080812 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46535
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:02.402423 9825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:02.402768 9825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:02.402957 9825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:02.410379 9825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:02.410509 9825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:02.410626 9825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:52:02.420907 9825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46535
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:02.422700 9825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:02.424305 9825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:02.437475 9831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:02.437443 9833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:02.437517 9830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:02.443697 9825 server_base.cc:1034] running on GCE node
I20260926 20:52:02.444345 9825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:02.445317 9825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:02.446535 9825 hybrid_clock.cc:648] HybridClock initialized: now 1790455922446487 us; error 47 us; skew 500 ppm
I20260926 20:52:02.446906 9825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:02 dist-test-slave-2dg3 krb5kdc[9719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455922, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:02.449980 9825 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:52:02.451546 9825 webserver.cc:460] Webserver started at http://127.8.249.129:46089/ using document root <none> and password file <none>
I20260926 20:52:02.452234 9825 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:02.452332 9825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:02.452584 9825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:02.455437 9825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "158dd205eefe403f8577daccd3b53845"
format_stamp: "Formatted at 2026-09-26 20:52:02 on dist-test-slave-2dg3"
server_key: "65dd3f0438d7ad668c1a93b552df8aac"
server_key_iv: "24894943984b97df07aa92fede79bf3d"
server_key_version: "encryptionkey@0"
I20260926 20:52:02.456159 9825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "158dd205eefe403f8577daccd3b53845"
format_stamp: "Formatted at 2026-09-26 20:52:02 on dist-test-slave-2dg3"
server_key: "65dd3f0438d7ad668c1a93b552df8aac"
server_key_iv: "24894943984b97df07aa92fede79bf3d"
server_key_version: "encryptionkey@0"
I20260926 20:52:02.461380 9825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 20:52:02.465015 9840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:02.466367 9825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:52:02.466538 9825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "158dd205eefe403f8577daccd3b53845"
format_stamp: "Formatted at 2026-09-26 20:52:02 on dist-test-slave-2dg3"
server_key: "65dd3f0438d7ad668c1a93b552df8aac"
server_key_iv: "24894943984b97df07aa92fede79bf3d"
server_key_version: "encryptionkey@0"
I20260926 20:52:02.466734 9825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:02.500710 9825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:02.504796 9825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:02.505105 9825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:02.506081 9825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:02.507465 9825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:02.507541 9825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:02.507706 9825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:02.507766 9825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:02.540568 9825 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:45281
I20260926 20:52:02.540613 9952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:45281 every 8 connection(s)
I20260926 20:52:02.541864 9825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:52:02.550218 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 9825
I20260926 20:52:02.550446 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:52:02.551113 9190 external_mini_cluster.cc:1428] Setting key 4ff7152e12fd874ca630b99f78f5a086
Sep 26 20:52:02 dist-test-slave-2dg3 krb5kdc[9719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455922, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:02.562402 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:02.544357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:57861 (local address 127.8.249.190:46535)
0926 20:52:02.544767 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:02.544771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:02.545795 (+ 1024us) server_negotiation.cc:407] Connection header received
0926 20:52:02.547762 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:02.547768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:02.547856 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:02.547970 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:02.549477 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:02.550519 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:02.552409 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:02.552906 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:02.553054 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:02.556454 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:02.556463 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:02.556471 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:02.556515 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:02.559269 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:02.560197 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:02.560201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:02.560204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:02.560320 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:02.560763 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:02.560767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:02.560769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:02.561014 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:02.561114 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:02.561994 (+ 880us) server_negotiation.cc:299] Negotiation successful
0926 20:52:02.562156 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":216,"threads_started":1}
I20260926 20:52:02.564082 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:02.544983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46535 (local address 127.8.249.129:57861)
0926 20:52:02.545639 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:02.545681 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:02.547492 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:02.548222 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:02.548230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:02.548749 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:02.549281 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:02.549296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:02.550660 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:02.550664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:02.552275 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:02.552290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:02.553140 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:02.553145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:02.553345 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:02.554041 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:02.554068 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:02.556218 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:02.559404 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:02.559416 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:02.559436 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:02.560081 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:02.560414 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:02.560419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:02.560422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:02.560662 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:02.561135 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:02.561145 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:02.561607 (+ 462us) client_negotiation.cc:769] Sending connection context
0926 20:52:02.561908 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 20:52:02.562515 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":248,"threads_started":1}
I20260926 20:52:02.565600 9953 heartbeater.cc:347] Connected to a master server at 127.8.249.190:46535
I20260926 20:52:02.565960 9953 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:02.566927 9953 heartbeater.cc:511] Master 127.8.249.190:46535 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
I20260926 20:52:02.569335 9764 ts_manager.cc:194] Registered new tserver with Master: 158dd205eefe403f8577daccd3b53845 (127.8.249.129:45281)
I20260926 20:52:02.570951 9764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:57861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:52:02.593317 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46535
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:02.915655 9959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:02.915940 9959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:02.916038 9959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:02.923264 9959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:02.923386 9959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:02.923511 9959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:52:02.933429 9959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46535
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:02.934697 9959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:02.936010 9959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:02.948901 9964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:02.948854 9965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:02.949095 9967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:02.950356 9959 server_base.cc:1034] running on GCE node
I20260926 20:52:02.951130 9959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:02.952162 9959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:02.953403 9959 hybrid_clock.cc:648] HybridClock initialized: now 1790455922953347 us; error 49 us; skew 500 ppm
I20260926 20:52:02.953744 9959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:02 dist-test-slave-2dg3 krb5kdc[9719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455922, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:02.956931 9959 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:52:02.958411 9959 webserver.cc:460] Webserver started at http://127.8.249.130:44207/ using document root <none> and password file <none>
I20260926 20:52:02.959190 9959 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:02.959291 9959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:02.959563 9959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:02.962448 9959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "c47172647cde4fe6a8b571ce2c7dbab5"
format_stamp: "Formatted at 2026-09-26 20:52:02 on dist-test-slave-2dg3"
server_key: "b78aed0fc4367d6572ea498fd7827967"
server_key_iv: "1f0fa4f26707f97b15e308cb45f8c77a"
server_key_version: "encryptionkey@0"
I20260926 20:52:02.963277 9959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "c47172647cde4fe6a8b571ce2c7dbab5"
format_stamp: "Formatted at 2026-09-26 20:52:02 on dist-test-slave-2dg3"
server_key: "b78aed0fc4367d6572ea498fd7827967"
server_key_iv: "1f0fa4f26707f97b15e308cb45f8c77a"
server_key_version: "encryptionkey@0"
I20260926 20:52:02.968741 9959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:52:02.972528 9974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:02.973896 9959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:52:02.974028 9959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "c47172647cde4fe6a8b571ce2c7dbab5"
format_stamp: "Formatted at 2026-09-26 20:52:02 on dist-test-slave-2dg3"
server_key: "b78aed0fc4367d6572ea498fd7827967"
server_key_iv: "1f0fa4f26707f97b15e308cb45f8c77a"
server_key_version: "encryptionkey@0"
I20260926 20:52:02.974200 9959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:02.993654 9959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:02.997699 9959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:02.997946 9959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:02.998901 9959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:03.000211 9959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:03.000281 9959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:03.000377 9959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:03.000420 9959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:03.032449 9959 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:44461
I20260926 20:52:03.032588 10086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:44461 every 8 connection(s)
I20260926 20:52:03.033720 9959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:52:03.040776 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 9959
I20260926 20:52:03.040992 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:52:03.041535 9190 external_mini_cluster.cc:1428] Setting key 9da0c725ee1c574f58c063a5fda8534d
Sep 26 20:52:03 dist-test-slave-2dg3 krb5kdc[9719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455922, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:03.055023 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.036132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:37435 (local address 127.8.249.190:46535)
0926 20:52:03.036345 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:03.036351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:03.037553 (+ 1202us) server_negotiation.cc:407] Connection header received
0926 20:52:03.039401 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:03.039406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:03.039476 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:03.039558 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:03.041439 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.042285 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.044332 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.044739 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.044857 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.049246 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:03.049251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:03.049254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:03.049293 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:03.051893 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.052694 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.052699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.052701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.052825 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.053288 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.053292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.053294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.053550 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:03.053701 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:03.054422 (+ 721us) server_negotiation.cc:299] Negotiation successful
0926 20:52:03.054657 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:52:03.056020 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.036692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46535 (local address 127.8.249.130:37435)
0926 20:52:03.037381 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:03.037424 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:03.039122 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:03.039769 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:03.039777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:03.040345 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:03.041250 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.041276 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.042463 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.042468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.044180 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.044195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.045005 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.045015 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.045393 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.046193 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:03.046231 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:03.048988 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:03.052050 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:03.052059 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:03.052078 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:03.052561 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:03.052956 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:03.052962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:03.052965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:03.053161 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:03.053673 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:03.053681 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:03.054055 (+ 374us) client_negotiation.cc:769] Sending connection context
0926 20:52:03.054317 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 20:52:03.054915 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":261,"threads_started":1}
I20260926 20:52:03.057464 10087 heartbeater.cc:347] Connected to a master server at 127.8.249.190:46535
I20260926 20:52:03.057775 10087 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
I20260926 20:52:03.058524 10087 heartbeater.cc:511] Master 127.8.249.190:46535 requested a full tablet report, sending...
I20260926 20:52:03.059933 9764 ts_manager.cc:194] Registered new tserver with Master: c47172647cde4fe6a8b571ce2c7dbab5 (127.8.249.130:44461)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
I20260926 20:52:03.060738 9764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:37435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:52:03.081815 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46535
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:03.401439 10092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:03.401808 10092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:03.401911 10092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:03.409152 10092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:03.409271 10092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:03.409389 10092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:52:03.419346 10092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46535
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:03.420643 10092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:03.421978 10092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:03.436064 10097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:03.435956 10098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:03.436028 10100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:03.436977 10092 server_base.cc:1034] running on GCE node
I20260926 20:52:03.437561 10092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:03.438525 10092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:03.439715 10092 hybrid_clock.cc:648] HybridClock initialized: now 1790455923439672 us; error 45 us; skew 500 ppm
I20260926 20:52:03.439985 10092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:03 dist-test-slave-2dg3 krb5kdc[9719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455923, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:03.443116 10092 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:52:03.444594 10092 webserver.cc:460] Webserver started at http://127.8.249.131:44907/ using document root <none> and password file <none>
I20260926 20:52:03.445302 10092 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:03.445430 10092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:03.445741 10092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:03.448763 10092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "085fc4d1a6f440d18bc12a43c6189d28"
format_stamp: "Formatted at 2026-09-26 20:52:03 on dist-test-slave-2dg3"
server_key: "1c1edc208116303800ac1b84c339a508"
server_key_iv: "5533a4f6645eab7f5c96ebf7fa2bda04"
server_key_version: "encryptionkey@0"
I20260926 20:52:03.449525 10092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "085fc4d1a6f440d18bc12a43c6189d28"
format_stamp: "Formatted at 2026-09-26 20:52:03 on dist-test-slave-2dg3"
server_key: "1c1edc208116303800ac1b84c339a508"
server_key_iv: "5533a4f6645eab7f5c96ebf7fa2bda04"
server_key_version: "encryptionkey@0"
I20260926 20:52:03.454555 10092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:52:03.457935 10107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:03.459224 10092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:03.459398 10092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "085fc4d1a6f440d18bc12a43c6189d28"
format_stamp: "Formatted at 2026-09-26 20:52:03 on dist-test-slave-2dg3"
server_key: "1c1edc208116303800ac1b84c339a508"
server_key_iv: "5533a4f6645eab7f5c96ebf7fa2bda04"
server_key_version: "encryptionkey@0"
I20260926 20:52:03.459582 10092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:03.472836 10092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:03.476686 10092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:03.476959 10092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:03.477890 10092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:03.479238 10092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:03.479312 10092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:03.479422 10092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:03.479483 10092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:03.510782 10092 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:38751
I20260926 20:52:03.510859 10219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:38751 every 8 connection(s)
I20260926 20:52:03.512154 10092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:52:03.518260 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 10092
I20260926 20:52:03.518452 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:52:03.518925 9190 external_mini_cluster.cc:1428] Setting key 3634f60aab3c1a122a8631aee9138f22
Sep 26 20:52:03 dist-test-slave-2dg3 krb5kdc[9719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455923, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:03.532222 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.514574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:42647 (local address 127.8.249.190:46535)
0926 20:52:03.514751 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:03.514755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:03.515929 (+ 1174us) server_negotiation.cc:407] Connection header received
0926 20:52:03.517769 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:03.517775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:03.517904 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:03.518021 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:03.519816 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.520955 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.522742 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.523211 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.523308 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.526602 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:03.526607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:03.526610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:03.526643 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:03.529208 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.529927 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.529934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.529937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.530070 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.530493 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.530500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.530504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.530761 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:03.530941 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:03.531639 (+ 698us) server_negotiation.cc:299] Negotiation successful
0926 20:52:03.531865 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:52:03.533610 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.515095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46535 (local address 127.8.249.131:42647)
0926 20:52:03.515761 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:03.515806 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:03.517487 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:03.518228 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:03.518237 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:03.518907 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:03.519624 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.519654 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.521096 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.521102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.522622 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.522633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.523402 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.523407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.523620 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.524223 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:03.524245 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:03.526385 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:03.529357 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:03.529367 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:03.529385 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:03.529805 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:03.530178 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:03.530182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:03.530186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:03.530385 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:03.530915 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:03.530923 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:03.531277 (+ 354us) client_negotiation.cc:769] Sending connection context
0926 20:52:03.531558 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 20:52:03.532122 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":259,"threads_started":1}
I20260926 20:52:03.534994 10220 heartbeater.cc:347] Connected to a master server at 127.8.249.190:46535
I20260926 20:52:03.535284 10220 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:03.535976 10220 heartbeater.cc:511] Master 127.8.249.190:46535 requested a full tablet report, sending...
I20260926 20:52:03.537289 9764 ts_manager.cc:194] Registered new tserver with Master: 085fc4d1a6f440d18bc12a43c6189d28 (127.8.249.131:38751)
I20260926 20:52:03.538061 9764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:42647
I20260926 20:52:03.546186 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:52:03.564517 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.552179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.8.249.190:46535)
0926 20:52:03.552367 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:03.552372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:03.552874 (+ 502us) server_negotiation.cc:407] Connection header received
0926 20:52:03.553026 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:03.553029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:03.553093 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:03.553187 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:03.554617 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.555423 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.556649 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.557239 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.557351 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.559571 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:03.559576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:03.559579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:03.559613 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:03.562109 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.562773 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.562779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.562782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.562959 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.563305 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.563308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.563311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.563636 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:03.563854 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:03.564127 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 20:52:03.564289 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:52:03.573971 9953 heartbeater.cc:503] Master 127.8.249.190:46535 was elected leader, sending a full tablet report...
I20260926 20:52:03.570603 9764 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:52:03.578029 9764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:52:03.604224 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.590152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:38751 (local address 127.0.0.1:49256)
0926 20:52:03.591146 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:03.591206 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:03.591423 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:03.596739 (+ 5316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:03.596746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:03.596774 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:03.597169 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.597183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.599331 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.599337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.601684 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.601698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.603129 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.603135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.603360 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.603445 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 20:52:03.603577 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 20:52:03.603767 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":90,"thread_start_us":5872,"threads_started":1}
I20260926 20:52:03.604885 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.590737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.8.249.130:44461)
0926 20:52:03.595469 (+ 4732us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:03.595483 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:03.596437 (+ 954us) server_negotiation.cc:407] Connection header received
0926 20:52:03.596746 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:03.596754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:03.596981 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:03.597163 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:03.598125 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.599554 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.602097 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.603577 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.603709 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.603896 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:03.604412 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 20:52:03.604582 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4498,"thread_start_us":4281,"threads_started":1}
I20260926 20:52:03.605156 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.590639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.8.249.131:38751)
0926 20:52:03.596005 (+ 5366us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:03.596015 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:03.596055 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:52:03.596180 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:03.596190 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:03.596491 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:03.597375 (+ 884us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:03.597691 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.599166 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.601858 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.602941 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.603078 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.603326 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:03.604669 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0926 20:52:03.604823 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5189,"thread_start_us":5034,"threads_started":1}
I20260926 20:52:03.605638 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.590346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:44461 (local address 127.0.0.1:34302)
0926 20:52:03.596320 (+ 5974us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:03.596339 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:03.596555 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:03.597440 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:03.597445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:03.597479 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:03.597963 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.597973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.599727 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.599732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.601752 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.601764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.604044 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.604051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.604200 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.604234 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:52:03.605089 (+ 855us) client_negotiation.cc:240] Negotiation successful
0926 20:52:03.605235 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5836,"mutex_wait_us":119,"thread_start_us":621,"threads_started":1}
I20260926 20:52:03.606447 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.597315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:45281 (local address 127.0.0.1:53244)
0926 20:52:03.597967 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:03.597986 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:03.598110 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:03.598689 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:03.598694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:03.598717 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:03.599039 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.599049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.600979 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.600985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.603295 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.603311 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.605650 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.605654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.605842 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.605876 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:52:03.605974 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:52:03.606096 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":95,"thread_start_us":308,"threads_started":1}
I20260926 20:52:03.607177 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.596686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.8.249.129:45281)
0926 20:52:03.597302 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:03.597311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:03.598162 (+ 851us) server_negotiation.cc:407] Connection header received
0926 20:52:03.598262 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:03.598269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:03.598502 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:03.598686 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:03.599243 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.600830 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.603462 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.605230 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.605363 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.605534 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:03.606026 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 20:52:03.606887 (+ 861us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":315,"threads_started":1}
I20260926 20:52:03.608842 10022 tablet_service.cc:1467] Processing CreateTablet for tablet 776b852a3b964f82ac83ce12f677fb2f (DEFAULT_TABLE table=test-table [id=3ffb0036aac94d62a587835d1ba9165a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:03.608848 10155 tablet_service.cc:1467] Processing CreateTablet for tablet 776b852a3b964f82ac83ce12f677fb2f (DEFAULT_TABLE table=test-table [id=3ffb0036aac94d62a587835d1ba9165a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:03.610262 10022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776b852a3b964f82ac83ce12f677fb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:03.610360 10155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776b852a3b964f82ac83ce12f677fb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:03.612383 9888 tablet_service.cc:1467] Processing CreateTablet for tablet 776b852a3b964f82ac83ce12f677fb2f (DEFAULT_TABLE table=test-table [id=3ffb0036aac94d62a587835d1ba9165a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:03.613739 9888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776b852a3b964f82ac83ce12f677fb2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:03.621377 10238 tablet_bootstrap.cc:492] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5: Bootstrap starting.
I20260926 20:52:03.624760 10239 tablet_bootstrap.cc:492] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28: Bootstrap starting.
I20260926 20:52:03.626165 10238 tablet_bootstrap.cc:654] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:03.627575 10238 log.cc:824] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:03.627966 10240 tablet_bootstrap.cc:492] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845: Bootstrap starting.
I20260926 20:52:03.628530 10239 tablet_bootstrap.cc:654] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:03.629694 10239 log.cc:824] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:03.631963 10238 tablet_bootstrap.cc:492] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5: No bootstrap required, opened a new log
I20260926 20:52:03.632246 10238 ts_tablet_manager.cc:1397] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 20:52:03.632603 10240 tablet_bootstrap.cc:654] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:03.633059 10239 tablet_bootstrap.cc:492] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28: No bootstrap required, opened a new log
I20260926 20:52:03.633319 10239 ts_tablet_manager.cc:1397] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 20:52:03.633899 10240 log.cc:824] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:03.636895 10240 tablet_bootstrap.cc:492] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845: No bootstrap required, opened a new log
I20260926 20:52:03.637153 10240 ts_tablet_manager.cc:1397] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 20:52:03.640065 10238 raft_consensus.cc:348] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085fc4d1a6f440d18bc12a43c6189d28" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 38751 } } peers { permanent_uuid: "c47172647cde4fe6a8b571ce2c7dbab5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 44461 } } peers { permanent_uuid: "158dd205eefe403f8577daccd3b53845" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45281 } }
I20260926 20:52:03.640435 10238 raft_consensus.cc:374] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:03.640539 10238 raft_consensus.cc:729] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47172647cde4fe6a8b571ce2c7dbab5, State: Initialized, Role: FOLLOWER
I20260926 20:52:03.641125 10238 consensus_queue.cc:260] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085fc4d1a6f440d18bc12a43c6189d28" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 38751 } } peers { permanent_uuid: "c47172647cde4fe6a8b571ce2c7dbab5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 44461 } } peers { permanent_uuid: "158dd205eefe403f8577daccd3b53845" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45281 } }
I20260926 20:52:03.643700 10239 raft_consensus.cc:348] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085fc4d1a6f440d18bc12a43c6189d28" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 38751 } } peers { permanent_uuid: "c47172647cde4fe6a8b571ce2c7dbab5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 44461 } } peers { permanent_uuid: "158dd205eefe403f8577daccd3b53845" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45281 } }
I20260926 20:52:03.644048 10087 heartbeater.cc:503] Master 127.8.249.190:46535 was elected leader, sending a full tablet report...
I20260926 20:52:03.644167 10239 raft_consensus.cc:374] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:03.644273 10239 raft_consensus.cc:729] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085fc4d1a6f440d18bc12a43c6189d28, State: Initialized, Role: FOLLOWER
I20260926 20:52:03.644722 10238 ts_tablet_manager.cc:1428] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 20:52:03.644905 10239 consensus_queue.cc:260] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085fc4d1a6f440d18bc12a43c6189d28" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 38751 } } peers { permanent_uuid: "c47172647cde4fe6a8b571ce2c7dbab5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 44461 } } peers { permanent_uuid: "158dd205eefe403f8577daccd3b53845" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45281 } }
I20260926 20:52:03.645985 10220 heartbeater.cc:503] Master 127.8.249.190:46535 was elected leader, sending a full tablet report...
I20260926 20:52:03.646631 10239 ts_tablet_manager.cc:1428] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260926 20:52:03.649089 10240 raft_consensus.cc:348] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085fc4d1a6f440d18bc12a43c6189d28" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 38751 } } peers { permanent_uuid: "c47172647cde4fe6a8b571ce2c7dbab5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 44461 } } peers { permanent_uuid: "158dd205eefe403f8577daccd3b53845" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45281 } }
I20260926 20:52:03.649549 10240 raft_consensus.cc:374] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:03.649641 10240 raft_consensus.cc:729] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158dd205eefe403f8577daccd3b53845, State: Initialized, Role: FOLLOWER
I20260926 20:52:03.650249 10240 consensus_queue.cc:260] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085fc4d1a6f440d18bc12a43c6189d28" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 38751 } } peers { permanent_uuid: "c47172647cde4fe6a8b571ce2c7dbab5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 44461 } } peers { permanent_uuid: "158dd205eefe403f8577daccd3b53845" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45281 } }
I20260926 20:52:03.651691 10240 ts_tablet_manager.cc:1428] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260926 20:52:03.659437 10246 raft_consensus.cc:482] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:03.659667 10246 raft_consensus.cc:504] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085fc4d1a6f440d18bc12a43c6189d28" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 38751 } } peers { permanent_uuid: "c47172647cde4fe6a8b571ce2c7dbab5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 44461 } } peers { permanent_uuid: "158dd205eefe403f8577daccd3b53845" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45281 } }
I20260926 20:52:03.661268 10246 leader_election.cc:288] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 085fc4d1a6f440d18bc12a43c6189d28 (127.8.249.131:38751), c47172647cde4fe6a8b571ce2c7dbab5 (127.8.249.130:44461)
I20260926 20:52:03.668054 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.661737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:59773 (local address 127.8.249.130:44461)
0926 20:52:03.661936 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:03.661944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:03.662659 (+ 715us) server_negotiation.cc:407] Connection header received
0926 20:52:03.662902 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:03.662906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:03.662974 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:03.663095 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:03.663727 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.664550 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.666362 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.667182 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.667346 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.667421 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:03.667675 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 20:52:03.667790 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:52:03.668179 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.661473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:44461 (local address 127.8.249.129:59773)
0926 20:52:03.662541 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:03.662562 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:03.662734 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:03.663121 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:03.663126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:03.663153 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:03.663578 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.663596 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.664666 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.664674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.666221 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.666232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.667368 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.667374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.667531 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.667554 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:52:03.667652 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:52:03.667778 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":53,"thread_start_us":604,"threads_started":1}
I20260926 20:52:03.668777 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.661644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:39167 (local address 127.8.249.131:38751)
0926 20:52:03.661884 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:03.661892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:03.662736 (+ 844us) server_negotiation.cc:407] Connection header received
0926 20:52:03.663028 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:03.663036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:03.663133 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:03.663233 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:03.664041 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.664809 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.666711 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.667603 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.667790 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.667878 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:03.668364 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 20:52:03.668510 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:52:03.669126 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776b852a3b964f82ac83ce12f677fb2f" candidate_uuid: "158dd205eefe403f8577daccd3b53845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47172647cde4fe6a8b571ce2c7dbab5" is_pre_election: true
I20260926 20:52:03.669346 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.661910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:38751 (local address 127.8.249.129:39167)
0926 20:52:03.662622 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:03.662640 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:03.662762 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:03.663400 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:03.663407 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:03.663439 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:03.663895 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.663907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.664962 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.664969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.666548 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:03.666563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.667841 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.667848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:03.668020 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.668049 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 20:52:03.668960 (+ 911us) client_negotiation.cc:240] Negotiation successful
0926 20:52:03.669063 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":1,"thread_start_us":417,"threads_started":1}
I20260926 20:52:03.669620 10042 raft_consensus.cc:2454] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158dd205eefe403f8577daccd3b53845 in term 0.
I20260926 20:52:03.670212 9842 leader_election.cc:302] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158dd205eefe403f8577daccd3b53845, c47172647cde4fe6a8b571ce2c7dbab5; no voters:
I20260926 20:52:03.670301 10175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776b852a3b964f82ac83ce12f677fb2f" candidate_uuid: "158dd205eefe403f8577daccd3b53845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085fc4d1a6f440d18bc12a43c6189d28" is_pre_election: true
I20260926 20:52:03.670596 10246 raft_consensus.cc:2781] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:52:03.670763 10246 raft_consensus.cc:482] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:03.670893 10246 raft_consensus.cc:3037] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:03.670990 10175 raft_consensus.cc:2454] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158dd205eefe403f8577daccd3b53845 in term 0.
I20260926 20:52:03.673461 10246 raft_consensus.cc:504] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085fc4d1a6f440d18bc12a43c6189d28" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 38751 } } peers { permanent_uuid: "c47172647cde4fe6a8b571ce2c7dbab5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 44461 } } peers { permanent_uuid: "158dd205eefe403f8577daccd3b53845" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45281 } }
I20260926 20:52:03.674108 10246 leader_election.cc:288] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [CANDIDATE]: Term 1 election: Requested vote from peers 085fc4d1a6f440d18bc12a43c6189d28 (127.8.249.131:38751), c47172647cde4fe6a8b571ce2c7dbab5 (127.8.249.130:44461)
I20260926 20:52:03.674475 10175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776b852a3b964f82ac83ce12f677fb2f" candidate_uuid: "158dd205eefe403f8577daccd3b53845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "085fc4d1a6f440d18bc12a43c6189d28"
I20260926 20:52:03.674693 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776b852a3b964f82ac83ce12f677fb2f" candidate_uuid: "158dd205eefe403f8577daccd3b53845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c47172647cde4fe6a8b571ce2c7dbab5"
I20260926 20:52:03.674710 10175 raft_consensus.cc:3037] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:03.674922 10042 raft_consensus.cc:3037] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:03.677341 10175 raft_consensus.cc:2454] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158dd205eefe403f8577daccd3b53845 in term 1.
I20260926 20:52:03.677441 10042 raft_consensus.cc:2454] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158dd205eefe403f8577daccd3b53845 in term 1.
I20260926 20:52:03.677788 9841 leader_election.cc:302] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 085fc4d1a6f440d18bc12a43c6189d28, 158dd205eefe403f8577daccd3b53845; no voters:
I20260926 20:52:03.678097 10246 raft_consensus.cc:2781] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:03.678576 10246 raft_consensus.cc:686] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [term 1 LEADER]: Becoming Leader. State: Replica: 158dd205eefe403f8577daccd3b53845, State: Running, Role: LEADER
I20260926 20:52:03.679072 10246 consensus_queue.cc:237] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085fc4d1a6f440d18bc12a43c6189d28" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 38751 } } peers { permanent_uuid: "c47172647cde4fe6a8b571ce2c7dbab5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 44461 } } peers { permanent_uuid: "158dd205eefe403f8577daccd3b53845" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45281 } }
I20260926 20:52:03.683557 9764 catalog_manager.cc:5495] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845 reported cstate change: term changed from 0 to 1, leader changed from <none> to 158dd205eefe403f8577daccd3b53845 (127.8.249.129). New cstate: current_term: 1 leader_uuid: "158dd205eefe403f8577daccd3b53845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "085fc4d1a6f440d18bc12a43c6189d28" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c47172647cde4fe6a8b571ce2c7dbab5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 44461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "158dd205eefe403f8577daccd3b53845" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45281 } health_report { overall_health: HEALTHY } } }
Sep 26 20:52:03 dist-test-slave-2dg3 krb5kdc[9719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:52:03.706908 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.693329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53246 (local address 127.8.249.129:45281)
0926 20:52:03.693575 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:03.693584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:03.694007 (+ 423us) server_negotiation.cc:407] Connection header received
0926 20:52:03.694150 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:03.694154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:03.694223 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:03.694341 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:03.695812 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.696582 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.697797 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.698188 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.698291 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.702107 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:03.702124 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:03.702143 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:03.702185 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:03.704763 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.705327 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.705334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.705337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.705462 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.705756 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.705759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.705761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.706057 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:03.706259 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:03.706448 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 20:52:03.706603 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 20:52:03 dist-test-slave-2dg3 krb5kdc[9719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455923, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:52:03 dist-test-slave-2dg3 krb5kdc[9719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455923, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:52:03.750691 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.737579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53262 (local address 127.8.249.129:45281)
0926 20:52:03.737808 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:03.737817 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:03.738142 (+ 325us) server_negotiation.cc:407] Connection header received
0926 20:52:03.738354 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:03.738358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:03.738421 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:03.738529 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:03.739992 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.740724 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.741993 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.742417 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.742546 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.746244 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:03.746249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:03.746252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:03.746295 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:03.748690 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.749204 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.749207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.749210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.749326 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.749668 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.749672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.749674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.749927 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:03.750141 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:03.750300 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 20:52:03.750455 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 20:52:03.763764 10221 tablet.cc:2369] T 776b852a3b964f82ac83ce12f677fb2f P 085fc4d1a6f440d18bc12a43c6189d28: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 20:52:03 dist-test-slave-2dg3 krb5kdc[9719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260926 20:52:03.785650 10088 tablet.cc:2369] T 776b852a3b964f82ac83ce12f677fb2f P c47172647cde4fe6a8b571ce2c7dbab5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 20:52:03 dist-test-slave-2dg3 krb5kdc[9719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:52:03.794212 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.780739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.8.249.129:45281)
0926 20:52:03.781017 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:03.781025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:03.781304 (+ 279us) server_negotiation.cc:407] Connection header received
0926 20:52:03.781449 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:03.781453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:03.781529 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:03.781614 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:03.782910 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.783632 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.784773 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.785190 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.785281 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.789279 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:03.789285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:03.789289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:03.789324 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:03.791662 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.792343 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.792348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.792352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.792480 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.792832 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.792836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.792840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.793103 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:03.793362 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:03.793632 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 20:52:03.793864 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260926 20:52:03.794353 9954 tablet.cc:2369] T 776b852a3b964f82ac83ce12f677fb2f P 158dd205eefe403f8577daccd3b53845: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:52:03.796022 9868 tablet_service.cc:2908] Rejecting scan request for tablet 776b852a3b964f82ac83ce12f677fb2f: Uninitialized: clean time has not yet been initialized
I20260926 20:52:03.798734 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 9825
I20260926 20:52:03.810491 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.799707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.8.249.129:45281)
0926 20:52:03.799914 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:03.799922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:03.800175 (+ 253us) server_negotiation.cc:407] Connection header received
0926 20:52:03.800308 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:03.800311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:03.800377 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:03.800450 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:03.801526 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.802242 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.803485 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.803892 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.803983 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.806047 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:03.806052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:03.806055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:03.806083 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:03.808356 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.808947 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.808950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.808953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.809059 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.809337 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.809340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.809342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.809574 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:03.809743 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:03.809949 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 20:52:03.810129 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:52:03.811162 9948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:03.901396 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 9825
I20260926 20:52:03.918402 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 9959
Sep 26 20:52:03 dist-test-slave-2dg3 krb5kdc[9719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:52:03.932730 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:03.919283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.8.249.130:44461)
0926 20:52:03.919483 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:03.919488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:03.919510 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:52:03.919642 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:03.919648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:03.919722 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:03.919833 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:03.920976 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.921739 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.923115 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:03.923525 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:03.923654 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:03.927622 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:03.927636 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:03.927652 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:03.927684 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:03.930269 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.930983 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.930990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.930992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.931101 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:03.931454 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:03.931457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:03.931459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:03.931789 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:03.932012 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:03.932271 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 20:52:03.932449 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:52:03.933387 10082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:04.022161 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 9959
I20260926 20:52:04.037989 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 10092
Sep 26 20:52:04 dist-test-slave-2dg3 krb5kdc[9719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:52:04.052597 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:04.038760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.8.249.131:38751)
0926 20:52:04.038962 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:04.038970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:04.039040 (+ 70us) server_negotiation.cc:407] Connection header received
0926 20:52:04.039246 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:04.039250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:04.039310 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:04.039411 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:04.040721 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:04.041428 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:04.042722 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:04.043165 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:04.043290 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:04.047245 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:04.047260 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:04.047276 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:04.047308 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:04.050121 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:04.050704 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:04.050712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:04.050714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:04.051011 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:04.051333 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:04.051336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:04.051338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:04.051673 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:04.051885 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:04.052070 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 20:52:04.052223 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:52:04.053218 10215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:04.139863 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 10092
I20260926 20:52:04.155704 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 9732
I20260926 20:52:04.156821 9794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:04.262447 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 9732
2026-09-26T20:52:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4037 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:52:04 dist-test-slave-2dg3 krb5kdc[10268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:52:04 dist-test-slave-2dg3 krb5kdc[10268](info): set up 2 sockets
Sep 26 20:52:04 dist-test-slave-2dg3 krb5kdc[10268](info): commencing operation
krb5kdc: starting...
W20260926 20:52:05.338713 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:52:05 dist-test-slave-2dg3 krb5kdc[10268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:52:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:52:05.519413 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:35217
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:05.836396 10281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:05.836700 10281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:05.836817 10281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:05.844080 10281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:05.844187 10281 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:52:05.844228 10281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:05.844280 10281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:05.844372 10281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:05.844480 10281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:05.854368 10281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:35217
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:05.855723 10281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:05.857031 10281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:05.869431 10289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:05.869451 10286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:05.869447 10287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:05.870083 10281 server_base.cc:1034] running on GCE node
I20260926 20:52:05.871022 10281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:05.872385 10281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:05.873595 10281 hybrid_clock.cc:648] HybridClock initialized: now 1790455925873516 us; error 54 us; skew 500 ppm
I20260926 20:52:05.873912 10281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:05 dist-test-slave-2dg3 krb5kdc[10268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455925, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:05.876987 10281 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:52:05.878443 10281 webserver.cc:460] Webserver started at http://127.8.249.190:33871/ using document root <none> and password file <none>
I20260926 20:52:05.879220 10281 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:05.879310 10281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:05.879642 10281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:05.882673 10281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "ec41f31db26146acb61d64807902ecce"
format_stamp: "Formatted at 2026-09-26 20:52:05 on dist-test-slave-2dg3"
server_key: "39d9bc20d3246c2ecb17a9a0a9ab5d5b"
server_key_iv: "5bc8e7ad458aa1f0c7fc9efce05ec0a8"
server_key_version: "encryptionkey@0"
I20260926 20:52:05.883617 10281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "ec41f31db26146acb61d64807902ecce"
format_stamp: "Formatted at 2026-09-26 20:52:05 on dist-test-slave-2dg3"
server_key: "39d9bc20d3246c2ecb17a9a0a9ab5d5b"
server_key_iv: "5bc8e7ad458aa1f0c7fc9efce05ec0a8"
server_key_version: "encryptionkey@0"
I20260926 20:52:05.889222 10281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:52:05.893067 10296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:05.894454 10281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:52:05.894660 10281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "ec41f31db26146acb61d64807902ecce"
format_stamp: "Formatted at 2026-09-26 20:52:05 on dist-test-slave-2dg3"
server_key: "39d9bc20d3246c2ecb17a9a0a9ab5d5b"
server_key_iv: "5bc8e7ad458aa1f0c7fc9efce05ec0a8"
server_key_version: "encryptionkey@0"
I20260926 20:52:05.894958 10281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:05.948254 10281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:05.952307 10281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:05.952582 10281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:05.973860 10281 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:35217
I20260926 20:52:05.973892 10357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:35217 every 8 connection(s)
I20260926 20:52:05.975299 10281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:05.977635 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 10281
I20260926 20:52:05.977898 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:05.978735 9190 external_mini_cluster.cc:1428] Setting key 13f3960af90e4604e13d838a83817771
I20260926 20:52:05.980499 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:05.992738 10358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce: Bootstrap starting.
Sep 26 20:52:05 dist-test-slave-2dg3 krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:05.996204 10358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:05.997418 10358 log.cc:824] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:06.000872 10358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce: No bootstrap required, opened a new log
I20260926 20:52:06.002614 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:05.981158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.8.249.190:35217)
0926 20:52:05.981954 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:05.981966 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:05.982005 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:52:05.983930 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:05.983955 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:05.984403 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:05.984880 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:05.986435 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:05.987901 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:05.989189 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:05.989955 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:05.990182 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:05.994650 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:05.994659 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:05.994676 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:05.994724 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:05.998364 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:05.999346 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:05.999352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:05.999358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:05.999528 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:05.999930 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:05.999934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:05.999936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:06.000466 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:06.000698 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:06.001189 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 20:52:06.001586 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":385,"threads_started":1}
I20260926 20:52:06.009980 10358 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec41f31db26146acb61d64807902ecce" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 35217 } }
I20260926 20:52:06.010360 10358 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:06.010476 10358 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec41f31db26146acb61d64807902ecce, State: Initialized, Role: FOLLOWER
I20260926 20:52:06.011106 10358 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec41f31db26146acb61d64807902ecce" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 35217 } }
I20260926 20:52:06.011334 10358 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:06.011459 10358 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:06.011627 10358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:06.013933 10358 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec41f31db26146acb61d64807902ecce" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 35217 } }
I20260926 20:52:06.014499 10358 leader_election.cc:302] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec41f31db26146acb61d64807902ecce; no voters:
I20260926 20:52:06.015156 10358 leader_election.cc:288] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:06.015281 10363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:06.016013 10363 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [term 1 LEADER]: Becoming Leader. State: Replica: ec41f31db26146acb61d64807902ecce, State: Running, Role: LEADER
I20260926 20:52:06.016553 10363 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec41f31db26146acb61d64807902ecce" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 35217 } }
I20260926 20:52:06.017328 10358 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:06.019258 10364 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec41f31db26146acb61d64807902ecce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec41f31db26146acb61d64807902ecce" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 35217 } } }
I20260926 20:52:06.019629 10364 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:06.020453 10365 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec41f31db26146acb61d64807902ecce. Latest consensus state: current_term: 1 leader_uuid: "ec41f31db26146acb61d64807902ecce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec41f31db26146acb61d64807902ecce" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 35217 } } }
I20260926 20:52:06.020780 10365 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:06.021374 10369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:06.026094 10369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:06.036362 10369 catalog_manager.cc:1347] Generated new cluster ID: 0460cc03338f4c338afa70f735721d67
I20260926 20:52:06.036491 10369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:06.052425 10369 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:06.053755 10369 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:06.064271 10369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce: Generated new TSK 0
I20260926 20:52:06.065394 10369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:52:06.114189 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:35217
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:06.445779 10384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:06.446122 10384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:06.446290 10384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:06.453651 10384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:06.453804 10384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:06.453943 10384 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:52:06.454072 10384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:52:06.464037 10384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.249.190:35217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:06.465354 10384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:06.466678 10384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:06.479333 10392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:06.479409 10389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:06.479377 10390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:06.480118 10384 server_base.cc:1034] running on GCE node
I20260926 20:52:06.480752 10384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:06.481638 10384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:06.482904 10384 hybrid_clock.cc:648] HybridClock initialized: now 1790455926482829 us; error 47 us; skew 500 ppm
I20260926 20:52:06.483219 10384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:06 dist-test-slave-2dg3 krb5kdc[10268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455926, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:06.486303 10384 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:52:06.487846 10384 webserver.cc:460] Webserver started at http://127.8.249.129:33883/ using document root <none> and password file <none>
I20260926 20:52:06.488555 10384 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:06.488678 10384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:06.488991 10384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:06.491842 10384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "2332a57f502141f289dc5a59fc8b6afc"
format_stamp: "Formatted at 2026-09-26 20:52:06 on dist-test-slave-2dg3"
server_key: "9ac2cd8ce56d6d20c73e80797f79ac2e"
server_key_iv: "ab58ee6bb35c0e1061228e4cac0324a9"
server_key_version: "encryptionkey@0"
I20260926 20:52:06.492667 10384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "2332a57f502141f289dc5a59fc8b6afc"
format_stamp: "Formatted at 2026-09-26 20:52:06 on dist-test-slave-2dg3"
server_key: "9ac2cd8ce56d6d20c73e80797f79ac2e"
server_key_iv: "ab58ee6bb35c0e1061228e4cac0324a9"
server_key_version: "encryptionkey@0"
I20260926 20:52:06.497660 10384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:52:06.501282 10399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:06.502755 10384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:52:06.502962 10384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "2332a57f502141f289dc5a59fc8b6afc"
format_stamp: "Formatted at 2026-09-26 20:52:06 on dist-test-slave-2dg3"
server_key: "9ac2cd8ce56d6d20c73e80797f79ac2e"
server_key_iv: "ab58ee6bb35c0e1061228e4cac0324a9"
server_key_version: "encryptionkey@0"
I20260926 20:52:06.503232 10384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:06.531389 10384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:06.535398 10384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:06.535619 10384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:06.538079 10384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:06.538200 10384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:06.538327 10384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:06.538398 10384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:52:06 dist-test-slave-2dg3 krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455926, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:06.561822 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:06.538834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:55191 (local address 127.8.249.190:35217)
0926 20:52:06.539214 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:06.539219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:06.541038 (+ 1819us) server_negotiation.cc:407] Connection header received
0926 20:52:06.543128 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:06.543135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:06.543230 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:06.543360 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:06.546059 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:06.546776 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:06.549356 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:06.549961 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:06.550097 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:06.554723 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:06.554728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:06.554731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:06.554764 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:06.558182 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:06.559260 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:06.559267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:06.559271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:06.559437 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:06.560048 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:06.560053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:06.560057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:06.560357 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:06.560515 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:06.561235 (+ 720us) server_negotiation.cc:299] Negotiation successful
0926 20:52:06.561454 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":191,"threads_started":1}
I20260926 20:52:06.562991 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:06.540041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:35217 (local address 127.8.249.129:55191)
0926 20:52:06.540879 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:06.540916 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:06.542750 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:06.543699 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:06.543714 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:06.544756 (+ 1042us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:06.545822 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:06.545860 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:06.546977 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:06.546987 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:06.549201 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:06.549215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:06.550220 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:06.550228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:06.550441 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:06.551366 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:06.551387 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:06.554451 (+ 3064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:06.558338 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:06.558347 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:06.558370 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:06.559099 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:06.559610 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:06.559616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:06.559619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:06.559882 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:06.560499 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:06.560506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:06.560869 (+ 363us) client_negotiation.cc:769] Sending connection context
0926 20:52:06.561129 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 20:52:06.561695 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":389,"threads_started":1}
I20260926 20:52:06.584661 10384 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:34463
I20260926 20:52:06.584746 10519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:34463 every 8 connection(s)
I20260926 20:52:06.585943 10384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:52:06.587729 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:06.572385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:56479 (local address 127.8.249.190:35217)
0926 20:52:06.572607 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:06.572615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:06.572916 (+ 301us) server_negotiation.cc:407] Connection header received
0926 20:52:06.573136 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:06.573141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:06.573209 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:06.573310 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:06.574561 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:06.575675 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:06.577277 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:06.578045 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:06.578197 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:06.580995 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:06.581004 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:06.581010 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:06.581056 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:06.584800 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:06.585503 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:06.585509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:06.585512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:06.585649 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:06.586016 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:06.586024 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:06.586027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:06.586334 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:06.586509 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:06.587142 (+ 633us) server_negotiation.cc:299] Negotiation successful
0926 20:52:06.587383 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:52:06.588246 10520 heartbeater.cc:347] Connected to a master server at 127.8.249.190:35217
I20260926 20:52:06.588498 10520 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:06.588745 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:06.571925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:35217 (local address 127.8.249.129:56479)
0926 20:52:06.572801 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:06.572830 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:06.572980 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:06.573440 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:06.573445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:06.574138 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:06.574417 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:06.574426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:06.575825 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:06.575830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:06.577133 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:06.577147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:06.578284 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:06.578297 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:06.578571 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:06.579335 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:06.579362 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:06.580779 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:06.584974 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:06.584979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:06.584996 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:06.585368 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:06.585770 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:06.585774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:06.585783 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:06.585901 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:06.586490 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:06.586497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:06.586764 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 20:52:06.588025 (+ 1261us) client_negotiation.cc:240] Negotiation successful
0926 20:52:06.588375 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":679,"threads_started":1}
I20260926 20:52:06.589305 10520 heartbeater.cc:511] Master 127.8.249.190:35217 requested a full tablet report, sending...
I20260926 20:52:06.592465 10313 ts_manager.cc:194] Registered new tserver with Master: 2332a57f502141f289dc5a59fc8b6afc (127.8.249.129:34463)
I20260926 20:52:06.594753 10313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:55191
I20260926 20:52:06.596909 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 10384
I20260926 20:52:06.597146 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:52:06.597882 9190 external_mini_cluster.cc:1428] Setting key b0e8e7a6cf47470aed14aa5355538604
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:52:06.635943 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:35217
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:06.958693 10524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:06.958997 10524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:06.959182 10524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:06.966639 10524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:06.966777 10524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:06.966959 10524 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:52:06.967085 10524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:52:06.977274 10524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.249.190:35217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:06.978601 10524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:06.979971 10524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:06.993532 10529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:06.993729 10532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:06.993585 10530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:06.994848 10524 server_base.cc:1034] running on GCE node
I20260926 20:52:06.995525 10524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:06.996459 10524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:06.997674 10524 hybrid_clock.cc:648] HybridClock initialized: now 1790455926997611 us; error 34 us; skew 500 ppm
I20260926 20:52:06.997965 10524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:07 dist-test-slave-2dg3 krb5kdc[10268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455927, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:07.006301 10524 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:52:07.007956 10524 webserver.cc:460] Webserver started at http://127.8.249.130:34609/ using document root <none> and password file <none>
I20260926 20:52:07.008661 10524 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:07.008780 10524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:07.009095 10524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:07.012122 10524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "15606209d6c44d17a95c5c4c6008facf"
format_stamp: "Formatted at 2026-09-26 20:52:07 on dist-test-slave-2dg3"
server_key: "974759eed6bd10f3f1a070fbb9e3b7b3"
server_key_iv: "316a6860f722c7dd5e40bbb6674eec08"
server_key_version: "encryptionkey@0"
I20260926 20:52:07.012856 10524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "15606209d6c44d17a95c5c4c6008facf"
format_stamp: "Formatted at 2026-09-26 20:52:07 on dist-test-slave-2dg3"
server_key: "974759eed6bd10f3f1a070fbb9e3b7b3"
server_key_iv: "316a6860f722c7dd5e40bbb6674eec08"
server_key_version: "encryptionkey@0"
I20260926 20:52:07.017761 10524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:52:07.021262 10539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:07.022739 10524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:52:07.022933 10524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "15606209d6c44d17a95c5c4c6008facf"
format_stamp: "Formatted at 2026-09-26 20:52:07 on dist-test-slave-2dg3"
server_key: "974759eed6bd10f3f1a070fbb9e3b7b3"
server_key_iv: "316a6860f722c7dd5e40bbb6674eec08"
server_key_version: "encryptionkey@0"
I20260926 20:52:07.023133 10524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:07.053158 10524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:07.057730 10524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:07.057964 10524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:07.060688 10524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:07.060775 10524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:07.060925 10524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:07.060981 10524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:52:07 dist-test-slave-2dg3 krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455927, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:07.086097 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.061278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:56365 (local address 127.8.249.190:35217)
0926 20:52:07.061485 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:07.061491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:07.063019 (+ 1528us) server_negotiation.cc:407] Connection header received
0926 20:52:07.065698 (+ 2679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:07.065706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:07.065804 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:07.065884 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:07.068605 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.069640 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.071415 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.072145 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.072306 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.077800 (+ 5494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:07.077806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:07.077809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:07.077843 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:07.081799 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:07.083046 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:07.083054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:07.083059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:07.083240 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:07.084009 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:07.084014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:07.084018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:07.084441 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:07.084605 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:07.085540 (+ 935us) server_negotiation.cc:299] Negotiation successful
0926 20:52:07.085753 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 20:52:07.087364 10548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.061941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:35217 (local address 127.8.249.130:56365)
0926 20:52:07.062821 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:07.062878 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:07.065338 (+ 2460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:07.066549 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:07.066561 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:07.067426 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:07.068418 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.068449 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.069788 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.069793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.071230 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.071249 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.072408 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.072416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.072728 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.073582 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:07.073620 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:07.077495 (+ 3875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:07.082000 (+ 4505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:07.082011 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:07.082037 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:07.082860 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:07.083476 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:07.083484 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:07.083490 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:07.083831 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:07.084599 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:07.084610 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:07.085041 (+ 431us) client_negotiation.cc:769] Sending connection context
0926 20:52:07.085435 (+ 394us) client_negotiation.cc:240] Negotiation successful
0926 20:52:07.085995 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":385,"threads_started":1}
I20260926 20:52:07.107506 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.092683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:60457 (local address 127.8.249.190:35217)
0926 20:52:07.092922 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:07.092930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:07.093499 (+ 569us) server_negotiation.cc:407] Connection header received
0926 20:52:07.093701 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:07.093705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:07.093760 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:07.093835 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:07.095391 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.096295 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.098327 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.098854 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.098973 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.101576 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:07.101584 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:07.101591 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:07.101631 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:07.104475 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:07.105365 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:07.105368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:07.105370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:07.105485 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:07.105906 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:07.105909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:07.105911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:07.106194 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:07.106298 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:07.106972 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 20:52:07.107224 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:52:07.107950 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.092687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:35217 (local address 127.8.249.130:60457)
0926 20:52:07.093373 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:07.093398 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:07.093588 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:07.093894 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:07.093901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:07.094572 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:07.095167 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.095185 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.096454 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.096461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.098175 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.098188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.099044 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.099050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.099204 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.099938 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:07.099971 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:07.101369 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:07.104590 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:07.104598 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:07.104605 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:07.105249 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:07.105598 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:07.105605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:07.105612 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:07.105795 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:07.106304 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:07.106334 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:07.106603 (+ 269us) client_negotiation.cc:769] Sending connection context
0926 20:52:07.106786 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 20:52:07.107284 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":345,"threads_started":1}
I20260926 20:52:07.111804 10524 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:33505
I20260926 20:52:07.111815 10658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:33505 every 8 connection(s)
I20260926 20:52:07.113145 10524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:52:07.114184 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 10524
I20260926 20:52:07.114391 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:52:07.114943 9190 external_mini_cluster.cc:1428] Setting key bd6d73c4fc973ad9db8a5ad193c99d99
I20260926 20:52:07.115204 10659 heartbeater.cc:347] Connected to a master server at 127.8.249.190:35217
I20260926 20:52:07.115375 10659 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:07.116039 10659 heartbeater.cc:511] Master 127.8.249.190:35217 requested a full tablet report, sending...
I20260926 20:52:07.117375 10313 ts_manager.cc:194] Registered new tserver with Master: 15606209d6c44d17a95c5c4c6008facf (127.8.249.130:33505)
I20260926 20:52:07.118630 10313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:56365
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:52:07.154560 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:35217
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:07.476460 10663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:07.476780 10663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:07.476939 10663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:07.484226 10663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:07.484380 10663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:07.484520 10663 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:52:07.484654 10663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:52:07.494840 10663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.249.190:35217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:07.496181 10663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:07.497547 10663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:07.510617 10671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:07.510653 10668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:07.510628 10669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:07.511502 10663 server_base.cc:1034] running on GCE node
I20260926 20:52:07.512135 10663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:07.513139 10663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:07.514380 10663 hybrid_clock.cc:648] HybridClock initialized: now 1790455927514306 us; error 50 us; skew 500 ppm
I20260926 20:52:07.514693 10663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:07 dist-test-slave-2dg3 krb5kdc[10268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455927, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:07.517860 10663 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:52:07.519416 10663 webserver.cc:460] Webserver started at http://127.8.249.131:34833/ using document root <none> and password file <none>
I20260926 20:52:07.520134 10663 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:07.520262 10663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:07.520586 10663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:07.523511 10663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "75eefe06349c4a1785ccbc4be557aa39"
format_stamp: "Formatted at 2026-09-26 20:52:07 on dist-test-slave-2dg3"
server_key: "b537c387a314f8423589c7db9b8dbe23"
server_key_iv: "3f79db9a9dbeec9a4b60465954579331"
server_key_version: "encryptionkey@0"
I20260926 20:52:07.524290 10663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "75eefe06349c4a1785ccbc4be557aa39"
format_stamp: "Formatted at 2026-09-26 20:52:07 on dist-test-slave-2dg3"
server_key: "b537c387a314f8423589c7db9b8dbe23"
server_key_iv: "3f79db9a9dbeec9a4b60465954579331"
server_key_version: "encryptionkey@0"
I20260926 20:52:07.529850 10663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:52:07.533629 10678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:07.535111 10663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:07.535293 10663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "75eefe06349c4a1785ccbc4be557aa39"
format_stamp: "Formatted at 2026-09-26 20:52:07 on dist-test-slave-2dg3"
server_key: "b537c387a314f8423589c7db9b8dbe23"
server_key_iv: "3f79db9a9dbeec9a4b60465954579331"
server_key_version: "encryptionkey@0"
I20260926 20:52:07.535527 10663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:07.558545 10663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:07.562677 10663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:07.562987 10663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:07.565649 10663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:07.565809 10663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:07.565937 10663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:07.566009 10663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:52:07 dist-test-slave-2dg3 krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455927, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:07.592006 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.567173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:35217 (local address 127.8.249.131:38767)
0926 20:52:07.568028 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:07.568067 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:07.570100 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:07.571059 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:07.571077 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:07.571932 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:07.572845 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.572876 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.574283 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.574288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.576484 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.576496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.577429 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.577439 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.577804 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.578756 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:07.578831 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:07.582239 (+ 3408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:07.586576 (+ 4337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:07.586587 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:07.586604 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:07.587336 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:07.587843 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:07.587848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:07.587851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:07.588163 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:07.588924 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:07.588932 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:07.589444 (+ 512us) client_negotiation.cc:769] Sending connection context
0926 20:52:07.589752 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 20:52:07.590371 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":363,"threads_started":1}
I20260926 20:52:07.593390 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.566316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:38767 (local address 127.8.249.190:35217)
0926 20:52:07.566543 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:07.566551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:07.568191 (+ 1640us) server_negotiation.cc:407] Connection header received
0926 20:52:07.570493 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:07.570503 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:07.570617 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:07.570758 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:07.573054 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.574107 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.576628 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.577144 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.577303 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.582555 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:07.582564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:07.582570 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:07.582623 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:07.586387 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:07.587516 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:07.587521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:07.587525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:07.587692 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:07.588303 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:07.588311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:07.588317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:07.588689 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:07.588832 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:07.592761 (+ 3929us) server_negotiation.cc:299] Negotiation successful
0926 20:52:07.593083 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 20:52:07.598367 10520 heartbeater.cc:503] Master 127.8.249.190:35217 was elected leader, sending a full tablet report...
I20260926 20:52:07.618180 10663 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:41503
I20260926 20:52:07.618155 10797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:41503 every 8 connection(s)
I20260926 20:52:07.619464 10663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:52:07.620136 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.606760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:38861 (local address 127.8.249.190:35217)
0926 20:52:07.606989 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:07.606998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:07.607997 (+ 999us) server_negotiation.cc:407] Connection header received
0926 20:52:07.608197 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:07.608202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:07.608278 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:07.608378 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:07.609306 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.610052 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.611721 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.612195 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.612319 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.614124 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:07.614130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:07.614135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:07.614173 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:07.617266 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:07.618112 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:07.618120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:07.618124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:07.618315 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:07.618679 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:07.618687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:07.618690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:07.619019 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:07.619169 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:07.619559 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 20:52:07.619779 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 20:52:07.620294 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.607245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:35217 (local address 127.8.249.131:38861)
0926 20:52:07.607882 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:07.607903 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:07.608065 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:07.608415 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:07.608420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:07.608879 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:07.609160 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.609169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.610187 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.610192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.611578 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.611592 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.612394 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.612401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.612566 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.613208 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:07.613226 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:07.613944 (+ 718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:07.617403 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:07.617410 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:07.617416 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:07.617963 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:07.618452 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:07.618455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:07.618458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:07.618573 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:07.619133 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:07.619137 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:07.619322 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 20:52:07.619465 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 20:52:07.619799 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":356,"threads_started":1}
I20260926 20:52:07.621599 10798 heartbeater.cc:347] Connected to a master server at 127.8.249.190:35217
I20260926 20:52:07.621865 10798 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:07.622316 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 10663
I20260926 20:52:07.622531 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:52:07.622701 10798 heartbeater.cc:511] Master 127.8.249.190:35217 requested a full tablet report, sending...
I20260926 20:52:07.623111 9190 external_mini_cluster.cc:1428] Setting key 9f1de9ad893ed2681fa3edf1b1a79409
I20260926 20:52:07.624193 10313 ts_manager.cc:194] Registered new tserver with Master: 75eefe06349c4a1785ccbc4be557aa39 (127.8.249.131:41503)
I20260926 20:52:07.625340 10313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:38767
I20260926 20:52:07.626109 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:52:07 dist-test-slave-2dg3 krb5kdc[10268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455927, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:52:07 dist-test-slave-2dg3 krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455927, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:07.671947 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.658480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.8.249.190:35217)
0926 20:52:07.658714 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:07.658723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:07.658882 (+ 159us) server_negotiation.cc:407] Connection header received
0926 20:52:07.659062 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:07.659067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:07.659139 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:07.659218 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:07.660621 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.661309 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.662518 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.663005 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.663100 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.666879 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:07.666883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:07.666886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:07.666921 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:07.669662 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:07.670256 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:07.670260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:07.670262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:07.670369 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:07.670658 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:07.670661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:07.670663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:07.670986 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:07.671164 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:07.671401 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 20:52:07.671565 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:52:07.675966 10312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:52:07.682535 10312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:52:07.707316 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.694350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:34463 (local address 127.0.0.1:44546)
0926 20:52:07.695453 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:07.695512 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:07.695755 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:07.697170 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:07.697181 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:07.697219 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:07.697744 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.697766 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.700491 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.700501 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.703703 (+ 3202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.703728 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.706316 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.706322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.706542 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.706626 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 20:52:07.706738 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 20:52:07.706954 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":1368,"threads_started":1}
I20260926 20:52:07.708505 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.694554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.8.249.129:34463)
0926 20:52:07.696384 (+ 1830us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:07.696405 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:07.696436 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:52:07.696565 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:07.696577 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:07.696891 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:07.697997 (+ 1106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:07.698239 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.700273 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.704051 (+ 3778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.706072 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.707696 (+ 1624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.707963 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:07.708043 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:52:07.708213 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1653,"thread_start_us":1518,"threads_started":1}
I20260926 20:52:07.712585 10450 tablet_service.cc:1467] Processing CreateTablet for tablet 43b97de01a5c4ac3a1614705c6ae4a32 (DEFAULT_TABLE table=test-table [id=047ab382b6164dc48d47536a447cd6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:07.713913 10450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b97de01a5c4ac3a1614705c6ae4a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:07.716423 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.696627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.8.249.130:33505)
0926 20:52:07.708126 (+ 11499us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:07.708133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:07.708159 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:52:07.708248 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:07.708255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:07.708499 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:07.709117 (+ 618us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:07.709944 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.711415 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.713977 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.715355 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.715515 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.715717 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:07.715962 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 20:52:07.716142 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11345,"thread_start_us":11202,"threads_started":1}
I20260926 20:52:07.716475 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.696145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:33505 (local address 127.0.0.1:37366)
0926 20:52:07.705919 (+ 9774us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:07.705941 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:07.706083 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:07.708726 (+ 2643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:07.708732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:07.708795 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:07.709736 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.709749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.711553 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.711563 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.713806 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.713817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.715581 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.715588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.715756 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.715789 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 20:52:07.715892 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:52:07.716030 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9618,"mutex_wait_us":511,"spinlock_wait_cycles":12288,"thread_start_us":10033,"threads_started":1}
I20260926 20:52:07.717700 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.694538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:41503 (local address 127.0.0.1:44512)
0926 20:52:07.696095 (+ 1557us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:07.696114 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:07.696228 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:07.708724 (+ 12496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:07.708731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:07.708788 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:07.709256 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.709274 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.711433 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.711441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.714705 (+ 3264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.714716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.716798 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.716804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.716996 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.717032 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 20:52:07.717132 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 20:52:07.717276 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1387,"mutex_wait_us":67,"spinlock_wait_cycles":17536,"thread_start_us":373,"threads_started":1}
I20260926 20:52:07.718349 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.695136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.8.249.131:41503)
0926 20:52:07.707011 (+ 11875us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:07.707024 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:07.707068 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:52:07.707187 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:07.707195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:07.708139 (+ 944us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:07.708356 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:07.709837 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.711300 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.714881 (+ 3581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.716117 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.716245 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.716466 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:07.717190 (+ 724us) server_negotiation.cc:299] Negotiation successful
0926 20:52:07.717414 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11632,"thread_start_us":12235,"threads_started":1}
I20260926 20:52:07.720186 10589 tablet_service.cc:1467] Processing CreateTablet for tablet 43b97de01a5c4ac3a1614705c6ae4a32 (DEFAULT_TABLE table=test-table [id=047ab382b6164dc48d47536a447cd6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:07.721643 10589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b97de01a5c4ac3a1614705c6ae4a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:07.722273 10728 tablet_service.cc:1467] Processing CreateTablet for tablet 43b97de01a5c4ac3a1614705c6ae4a32 (DEFAULT_TABLE table=test-table [id=047ab382b6164dc48d47536a447cd6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:07.723930 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b97de01a5c4ac3a1614705c6ae4a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:07.726847 10817 tablet_bootstrap.cc:492] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc: Bootstrap starting.
I20260926 20:52:07.730365 10817 tablet_bootstrap.cc:654] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:07.731596 10817 log.cc:824] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:07.736151 10819 tablet_bootstrap.cc:492] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf: Bootstrap starting.
I20260926 20:52:07.737403 10817 tablet_bootstrap.cc:492] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc: No bootstrap required, opened a new log
I20260926 20:52:07.737676 10817 ts_tablet_manager.cc:1397] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260926 20:52:07.740221 10819 tablet_bootstrap.cc:654] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:07.741402 10819 log.cc:824] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:07.742771 10820 tablet_bootstrap.cc:492] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39: Bootstrap starting.
I20260926 20:52:07.744351 10819 tablet_bootstrap.cc:492] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf: No bootstrap required, opened a new log
I20260926 20:52:07.744612 10819 ts_tablet_manager.cc:1397] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 20:52:07.746188 10820 tablet_bootstrap.cc:654] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:07.747406 10820 log.cc:824] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:07.748245 10817 raft_consensus.cc:348] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } }
I20260926 20:52:07.748595 10817 raft_consensus.cc:374] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:07.748701 10817 raft_consensus.cc:729] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2332a57f502141f289dc5a59fc8b6afc, State: Initialized, Role: FOLLOWER
I20260926 20:52:07.749322 10817 consensus_queue.cc:260] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } }
I20260926 20:52:07.750444 10820 tablet_bootstrap.cc:492] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39: No bootstrap required, opened a new log
I20260926 20:52:07.750706 10820 ts_tablet_manager.cc:1397] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 20:52:07.751173 10817 ts_tablet_manager.cc:1428] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260926 20:52:07.753295 10819 raft_consensus.cc:348] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } }
I20260926 20:52:07.753670 10819 raft_consensus.cc:374] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:07.753787 10819 raft_consensus.cc:729] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15606209d6c44d17a95c5c4c6008facf, State: Initialized, Role: FOLLOWER
I20260926 20:52:07.754431 10819 consensus_queue.cc:260] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } }
I20260926 20:52:07.755568 10659 heartbeater.cc:503] Master 127.8.249.190:35217 was elected leader, sending a full tablet report...
I20260926 20:52:07.756162 10819 ts_tablet_manager.cc:1428] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf: Time spent starting tablet: real 0.011s user 0.005s sys 0.007s
I20260926 20:52:07.761633 10820 raft_consensus.cc:348] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } }
I20260926 20:52:07.762012 10820 raft_consensus.cc:374] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:07.762121 10820 raft_consensus.cc:729] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75eefe06349c4a1785ccbc4be557aa39, State: Initialized, Role: FOLLOWER
I20260926 20:52:07.762693 10820 consensus_queue.cc:260] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } }
I20260926 20:52:07.763751 10798 heartbeater.cc:503] Master 127.8.249.190:35217 was elected leader, sending a full tablet report...
I20260926 20:52:07.764475 10820 ts_tablet_manager.cc:1428] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260926 20:52:07.807057 10825 raft_consensus.cc:482] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:07.807374 10825 raft_consensus.cc:504] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } }
I20260926 20:52:07.809051 10825 leader_election.cc:288] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2332a57f502141f289dc5a59fc8b6afc (127.8.249.129:34463), 15606209d6c44d17a95c5c4c6008facf (127.8.249.130:33505)
I20260926 20:52:07.814510 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.809352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:56335 (local address 127.8.249.129:34463)
0926 20:52:07.809539 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:07.809545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:07.809642 (+ 97us) server_negotiation.cc:407] Connection header received
0926 20:52:07.809858 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:07.809866 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:07.809944 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:07.810057 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:07.810474 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.811350 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.812881 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.813646 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.813814 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.813892 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:07.814157 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 20:52:07.814278 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:52:07.814512 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.809342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:34463 (local address 127.8.249.131:56335)
0926 20:52:07.809540 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:07.809560 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:07.809688 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:07.810075 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:07.810079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:07.810117 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:07.810335 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.810344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.811478 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.811486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.812722 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.812733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.813790 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.813796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.813994 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.814018 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:52:07.814118 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 20:52:07.814244 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:52:07.815593 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.809709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:33505 (local address 127.8.249.131:33627)
0926 20:52:07.810600 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:07.810634 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:07.810759 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:07.811089 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:07.811093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:07.811117 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:07.811424 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.811432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.812435 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.812441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.813727 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:07.813738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.815076 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.815081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:07.815191 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.815213 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:52:07.815287 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:52:07.815369 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":596,"threads_started":1}
I20260926 20:52:07.815459 10472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b97de01a5c4ac3a1614705c6ae4a32" candidate_uuid: "75eefe06349c4a1785ccbc4be557aa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2332a57f502141f289dc5a59fc8b6afc" is_pre_election: true
I20260926 20:52:07.815717 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.809838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:33627 (local address 127.8.249.130:33505)
0926 20:52:07.810092 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:07.810101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:07.810719 (+ 618us) server_negotiation.cc:407] Connection header received
0926 20:52:07.810900 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:07.810905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:07.810980 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:07.811073 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:07.811598 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.812303 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.813878 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.814696 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.814864 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.814929 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:07.815339 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 20:52:07.815470 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:52:07.816057 10472 raft_consensus.cc:2454] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75eefe06349c4a1785ccbc4be557aa39 in term 0.
I20260926 20:52:07.816681 10682 leader_election.cc:302] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2332a57f502141f289dc5a59fc8b6afc, 75eefe06349c4a1785ccbc4be557aa39; no voters:
I20260926 20:52:07.816648 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b97de01a5c4ac3a1614705c6ae4a32" candidate_uuid: "75eefe06349c4a1785ccbc4be557aa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15606209d6c44d17a95c5c4c6008facf" is_pre_election: true
I20260926 20:52:07.817097 10825 raft_consensus.cc:2781] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:52:07.817211 10825 raft_consensus.cc:482] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:07.817150 10609 raft_consensus.cc:2454] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75eefe06349c4a1785ccbc4be557aa39 in term 0.
I20260926 20:52:07.817314 10825 raft_consensus.cc:3037] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:07.819798 10825 raft_consensus.cc:504] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } }
I20260926 20:52:07.820375 10825 leader_election.cc:288] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [CANDIDATE]: Term 1 election: Requested vote from peers 2332a57f502141f289dc5a59fc8b6afc (127.8.249.129:34463), 15606209d6c44d17a95c5c4c6008facf (127.8.249.130:33505)
I20260926 20:52:07.820778 10472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b97de01a5c4ac3a1614705c6ae4a32" candidate_uuid: "75eefe06349c4a1785ccbc4be557aa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2332a57f502141f289dc5a59fc8b6afc"
I20260926 20:52:07.820842 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b97de01a5c4ac3a1614705c6ae4a32" candidate_uuid: "75eefe06349c4a1785ccbc4be557aa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15606209d6c44d17a95c5c4c6008facf"
I20260926 20:52:07.820958 10472 raft_consensus.cc:3037] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:07.821086 10609 raft_consensus.cc:3037] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:07.823215 10472 raft_consensus.cc:2454] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75eefe06349c4a1785ccbc4be557aa39 in term 1.
I20260926 20:52:07.823346 10609 raft_consensus.cc:2454] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75eefe06349c4a1785ccbc4be557aa39 in term 1.
I20260926 20:52:07.823598 10682 leader_election.cc:302] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2332a57f502141f289dc5a59fc8b6afc, 75eefe06349c4a1785ccbc4be557aa39; no voters:
I20260926 20:52:07.823935 10825 raft_consensus.cc:2781] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:07.824380 10825 raft_consensus.cc:686] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [term 1 LEADER]: Becoming Leader. State: Replica: 75eefe06349c4a1785ccbc4be557aa39, State: Running, Role: LEADER
I20260926 20:52:07.824849 10825 consensus_queue.cc:237] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } }
I20260926 20:52:07.829676 10313 catalog_manager.cc:5495] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75eefe06349c4a1785ccbc4be557aa39 (127.8.249.131). New cstate: current_term: 1 leader_uuid: "75eefe06349c4a1785ccbc4be557aa39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } health_report { overall_health: UNKNOWN } } }
W20260926 20:52:07.839111 10521 tablet.cc:2369] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:52:07.865054 10660 tablet.cc:2369] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:52:07.871783 10799 tablet.cc:2369] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:52:07.878399 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:07.872916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.8.249.131:41503)
0926 20:52:07.873124 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:07.873131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:07.873241 (+ 110us) server_negotiation.cc:407] Connection header received
0926 20:52:07.873586 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:07.873591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:07.873670 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:07.873823 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:07.874204 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.875006 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.876545 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:07.876941 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:07.877036 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:07.877466 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:07.877927 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:07.878036 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:07.878146 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 20:52:07.878210 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:52:07.885914 10609 raft_consensus.cc:1261] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf [term 1 FOLLOWER]: Refusing update from remote peer 75eefe06349c4a1785ccbc4be557aa39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:07.886098 10472 raft_consensus.cc:1261] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc [term 1 FOLLOWER]: Refusing update from remote peer 75eefe06349c4a1785ccbc4be557aa39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:07.886910 10825 consensus_queue.cc:1035] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:07.887219 10827 consensus_queue.cc:1035] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:07.901540 10831 mvcc.cc:205] Tried to move back new op lower bound from 7333707480611311616 to 7333707480375799808. Current Snapshot: MvccSnapshot[applied={T|T < 7333707480611311616}]
I20260926 20:52:07.919063 10312 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:07.919375 10312 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:07.923089 10312 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce: Sending DeleteTablet for 3 replicas of tablet 43b97de01a5c4ac3a1614705c6ae4a32
I20260926 20:52:07.924316 10589 tablet_service.cc:1514] Processing DeleteTablet for tablet 43b97de01a5c4ac3a1614705c6ae4a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:07 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:37366
I20260926 20:52:07.924319 10728 tablet_service.cc:1514] Processing DeleteTablet for tablet 43b97de01a5c4ac3a1614705c6ae4a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:07 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:44512
I20260926 20:52:07.925299 10450 tablet_service.cc:1514] Processing DeleteTablet for tablet 43b97de01a5c4ac3a1614705c6ae4a32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:07 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:44546
I20260926 20:52:07.925761 10313 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:52:07.926553 10313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:52:07.929620 10842 tablet_replica.cc:330] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc: stopping tablet replica
I20260926 20:52:07.930150 10842 raft_consensus.cc:2229] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:07.930807 10842 raft_consensus.cc:2258] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:07.931696 10840 tablet_replica.cc:330] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf: stopping tablet replica
I20260926 20:52:07.932106 10840 raft_consensus.cc:2229] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:07.932550 10840 raft_consensus.cc:2258] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:07.933439 10841 tablet_replica.cc:330] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39: stopping tablet replica
I20260926 20:52:07.933804 10841 raft_consensus.cc:2229] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:52:07.934172 10842 ts_tablet_manager.cc:1905] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:07.934427 10841 raft_consensus.cc:2258] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:07.935983 10840 ts_tablet_manager.cc:1905] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:07.936214 10450 tablet_service.cc:1467] Processing CreateTablet for tablet 9e36c73480544f069b52c6e63354bf27 (DEFAULT_TABLE table=test-table [id=a679bd8dbfcc48b5b51c0b12cdea46bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:07.936666 10450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e36c73480544f069b52c6e63354bf27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:07.938954 10728 tablet_service.cc:1467] Processing CreateTablet for tablet 9e36c73480544f069b52c6e63354bf27 (DEFAULT_TABLE table=test-table [id=a679bd8dbfcc48b5b51c0b12cdea46bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:07.939877 10589 tablet_service.cc:1467] Processing CreateTablet for tablet 9e36c73480544f069b52c6e63354bf27 (DEFAULT_TABLE table=test-table [id=a679bd8dbfcc48b5b51c0b12cdea46bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:07.940366 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e36c73480544f069b52c6e63354bf27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:07.941134 10842 ts_tablet_manager.cc:1918] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:07.941277 10842 log.cc:1196] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/43b97de01a5c4ac3a1614705c6ae4a32
I20260926 20:52:07.941780 10842 ts_tablet_manager.cc:1939] T 43b97de01a5c4ac3a1614705c6ae4a32 P 2332a57f502141f289dc5a59fc8b6afc: Deleting consensus metadata
I20260926 20:52:07.941845 10841 ts_tablet_manager.cc:1905] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:07.942963 10589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e36c73480544f069b52c6e63354bf27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:07.943674 10300 catalog_manager.cc:4841] TS 2332a57f502141f289dc5a59fc8b6afc (127.8.249.129:34463): tablet 43b97de01a5c4ac3a1614705c6ae4a32 (table test-table [id=047ab382b6164dc48d47536a447cd6f8]) successfully deleted
I20260926 20:52:07.949003 10841 ts_tablet_manager.cc:1918] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:07.949198 10841 log.cc:1196] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/43b97de01a5c4ac3a1614705c6ae4a32
I20260926 20:52:07.949811 10841 ts_tablet_manager.cc:1939] T 43b97de01a5c4ac3a1614705c6ae4a32 P 75eefe06349c4a1785ccbc4be557aa39: Deleting consensus metadata
I20260926 20:52:07.951539 10299 catalog_manager.cc:4841] TS 75eefe06349c4a1785ccbc4be557aa39 (127.8.249.131:41503): tablet 43b97de01a5c4ac3a1614705c6ae4a32 (table test-table [id=047ab382b6164dc48d47536a447cd6f8]) successfully deleted
I20260926 20:52:07.951843 10840 ts_tablet_manager.cc:1918] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:07.951957 10817 tablet_bootstrap.cc:492] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc: Bootstrap starting.
I20260926 20:52:07.952014 10840 log.cc:1196] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/43b97de01a5c4ac3a1614705c6ae4a32
I20260926 20:52:07.952652 10840 ts_tablet_manager.cc:1939] T 43b97de01a5c4ac3a1614705c6ae4a32 P 15606209d6c44d17a95c5c4c6008facf: Deleting consensus metadata
I20260926 20:52:07.953652 10817 tablet_bootstrap.cc:654] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:07.954027 10820 tablet_bootstrap.cc:492] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39: Bootstrap starting.
I20260926 20:52:07.954597 10300 catalog_manager.cc:4841] TS 15606209d6c44d17a95c5c4c6008facf (127.8.249.130:33505): tablet 43b97de01a5c4ac3a1614705c6ae4a32 (table test-table [id=047ab382b6164dc48d47536a447cd6f8]) successfully deleted
I20260926 20:52:07.955647 10820 tablet_bootstrap.cc:654] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:07.956707 10817 tablet_bootstrap.cc:492] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc: No bootstrap required, opened a new log
I20260926 20:52:07.956902 10817 ts_tablet_manager.cc:1397] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260926 20:52:07.958092 10817 raft_consensus.cc:348] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:07.958442 10817 raft_consensus.cc:374] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:07.958585 10817 raft_consensus.cc:729] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2332a57f502141f289dc5a59fc8b6afc, State: Initialized, Role: FOLLOWER
I20260926 20:52:07.959214 10819 tablet_bootstrap.cc:492] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf: Bootstrap starting.
I20260926 20:52:07.959869 10817 consensus_queue.cc:260] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:07.960645 10817 ts_tablet_manager.cc:1428] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260926 20:52:07.960886 10819 tablet_bootstrap.cc:654] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:07.962977 10820 tablet_bootstrap.cc:492] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39: No bootstrap required, opened a new log
I20260926 20:52:07.963193 10820 ts_tablet_manager.cc:1397] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260926 20:52:07.964109 10820 raft_consensus.cc:348] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:07.964406 10820 raft_consensus.cc:374] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:07.964560 10820 raft_consensus.cc:729] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75eefe06349c4a1785ccbc4be557aa39, State: Initialized, Role: FOLLOWER
I20260926 20:52:07.964928 10820 consensus_queue.cc:260] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:07.966295 10819 tablet_bootstrap.cc:492] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf: No bootstrap required, opened a new log
I20260926 20:52:07.966447 10819 ts_tablet_manager.cc:1397] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260926 20:52:07.965762 10820 ts_tablet_manager.cc:1428] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:52:07.967335 10819 raft_consensus.cc:348] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:07.967559 10819 raft_consensus.cc:374] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:07.967669 10819 raft_consensus.cc:729] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15606209d6c44d17a95c5c4c6008facf, State: Initialized, Role: FOLLOWER
I20260926 20:52:07.967967 10819 consensus_queue.cc:260] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:07.968544 10819 ts_tablet_manager.cc:1428] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 20:52:08.079493 10824 raft_consensus.cc:482] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:08.079743 10824 raft_consensus.cc:504] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:08.081298 10824 leader_election.cc:288] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2332a57f502141f289dc5a59fc8b6afc (127.8.249.129:34463), 75eefe06349c4a1785ccbc4be557aa39 (127.8.249.131:41503)
I20260926 20:52:08.089318 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.081806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:56027 (local address 127.8.249.131:41503)
0926 20:52:08.082007 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:08.082012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:08.083115 (+ 1103us) server_negotiation.cc:407] Connection header received
0926 20:52:08.083318 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:08.083328 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:08.083442 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:08.083529 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:08.084265 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.085295 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.087180 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.088196 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.088345 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.088436 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:08.088799 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 20:52:08.089000 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 20:52:08.089417 10847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.082194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:41503 (local address 127.8.249.130:56027)
0926 20:52:08.083032 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:08.083051 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:08.083196 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:08.083601 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:08.083607 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:08.083635 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:08.084091 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:08.084104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.085408 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.085416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:08.087031 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:08.087043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.088417 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.088424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:08.088618 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.088657 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 20:52:08.088769 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 20:52:08.088954 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":68,"thread_start_us":535,"threads_started":1}
I20260926 20:52:08.090430 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.081825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:35205 (local address 127.8.249.129:34463)
0926 20:52:08.082040 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:08.082047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:08.082906 (+ 859us) server_negotiation.cc:407] Connection header received
0926 20:52:08.083193 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:08.083201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:08.083285 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:08.083399 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:08.083987 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.085083 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.087815 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.088891 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.089045 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.089108 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:08.090032 (+ 924us) server_negotiation.cc:299] Negotiation successful
0926 20:52:08.090166 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 20:52:08.090270 10748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e36c73480544f069b52c6e63354bf27" candidate_uuid: "15606209d6c44d17a95c5c4c6008facf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75eefe06349c4a1785ccbc4be557aa39" is_pre_election: true
I20260926 20:52:08.090713 10748 raft_consensus.cc:2454] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15606209d6c44d17a95c5c4c6008facf in term 0.
I20260926 20:52:08.091197 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.082143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:34463 (local address 127.8.249.130:35205)
0926 20:52:08.082758 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:08.082782 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:08.083016 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:08.083437 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:08.083441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:08.083464 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:08.083825 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:08.083835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.085225 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.085231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:08.087648 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:08.087666 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.089279 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.089287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:08.089835 (+ 548us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.089869 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:52:08.090652 (+ 783us) client_negotiation.cc:240] Negotiation successful
0926 20:52:08.090817 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":84,"thread_start_us":304,"threads_started":1}
I20260926 20:52:08.091293 10542 leader_election.cc:302] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15606209d6c44d17a95c5c4c6008facf, 75eefe06349c4a1785ccbc4be557aa39; no voters:
I20260926 20:52:08.091711 10824 raft_consensus.cc:2781] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:52:08.091845 10824 raft_consensus.cc:482] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:08.091944 10824 raft_consensus.cc:3037] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:08.092119 10472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e36c73480544f069b52c6e63354bf27" candidate_uuid: "15606209d6c44d17a95c5c4c6008facf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2332a57f502141f289dc5a59fc8b6afc" is_pre_election: true
I20260926 20:52:08.092288 10472 raft_consensus.cc:2454] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15606209d6c44d17a95c5c4c6008facf in term 0.
I20260926 20:52:08.095201 10824 raft_consensus.cc:504] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:08.095990 10824 leader_election.cc:288] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [CANDIDATE]: Term 1 election: Requested vote from peers 2332a57f502141f289dc5a59fc8b6afc (127.8.249.129:34463), 75eefe06349c4a1785ccbc4be557aa39 (127.8.249.131:41503)
I20260926 20:52:08.096354 10472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e36c73480544f069b52c6e63354bf27" candidate_uuid: "15606209d6c44d17a95c5c4c6008facf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2332a57f502141f289dc5a59fc8b6afc"
I20260926 20:52:08.096493 10748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e36c73480544f069b52c6e63354bf27" candidate_uuid: "15606209d6c44d17a95c5c4c6008facf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75eefe06349c4a1785ccbc4be557aa39"
I20260926 20:52:08.096572 10472 raft_consensus.cc:3037] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:08.096716 10748 raft_consensus.cc:3037] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:08.098984 10748 raft_consensus.cc:2454] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15606209d6c44d17a95c5c4c6008facf in term 1.
I20260926 20:52:08.099330 10542 leader_election.cc:302] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15606209d6c44d17a95c5c4c6008facf, 75eefe06349c4a1785ccbc4be557aa39; no voters:
I20260926 20:52:08.099550 10472 raft_consensus.cc:2454] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15606209d6c44d17a95c5c4c6008facf in term 1.
I20260926 20:52:08.099596 10824 raft_consensus.cc:2781] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:08.100358 10824 raft_consensus.cc:686] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [term 1 LEADER]: Becoming Leader. State: Replica: 15606209d6c44d17a95c5c4c6008facf, State: Running, Role: LEADER
I20260926 20:52:08.100719 10824 consensus_queue.cc:237] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:08.104256 10312 catalog_manager.cc:5495] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf reported cstate change: term changed from 0 to 1, leader changed from <none> to 15606209d6c44d17a95c5c4c6008facf (127.8.249.130). New cstate: current_term: 1 leader_uuid: "15606209d6c44d17a95c5c4c6008facf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:52:08 dist-test-slave-2dg3 krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455925, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:08.178959 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.165261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.8.249.190:35217)
0926 20:52:08.165472 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:08.165482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:08.166113 (+ 631us) server_negotiation.cc:407] Connection header received
0926 20:52:08.166388 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:08.166393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:08.166480 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:08.166584 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:08.167796 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.168526 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.169751 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.170173 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.170307 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.173598 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:08.173604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:08.173609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:08.173644 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:08.176313 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:08.176970 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:08.176975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:08.176978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:08.177100 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:08.177447 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:08.177453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:08.177457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:08.177707 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:08.177848 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:08.178238 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 20:52:08.178573 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":63}
I20260926 20:52:08.178967 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.165292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:35217 (local address 127.0.0.1:50936)
0926 20:52:08.165961 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:08.165983 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:08.166133 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:08.166607 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:08.166612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:08.167359 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:08.167656 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:08.167666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.168651 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.168659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:08.169612 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:08.169623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.170347 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.170353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:08.170522 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.171243 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:08.171261 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:08.173423 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:08.176454 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:08.176466 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:08.176474 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:08.176846 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:08.177197 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:08.177204 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:08.177208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:08.177332 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:08.177823 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:08.177831 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:08.178001 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 20:52:08.178165 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 20:52:08.178454 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":385,"threads_started":1}
I20260926 20:52:08.184824 10312 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:50936:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:52:08.185838 10312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:52:08.193459 10450 tablet_service.cc:1467] Processing CreateTablet for tablet 742ff5c87be4406ca1148e7bc873d8b6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e95b16e9e8a24fd7b56d26a5f4e005ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:52:08.193535 10589 tablet_service.cc:1467] Processing CreateTablet for tablet 742ff5c87be4406ca1148e7bc873d8b6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e95b16e9e8a24fd7b56d26a5f4e005ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:52:08.193729 10728 tablet_service.cc:1467] Processing CreateTablet for tablet 742ff5c87be4406ca1148e7bc873d8b6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e95b16e9e8a24fd7b56d26a5f4e005ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:52:08.194130 10450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 742ff5c87be4406ca1148e7bc873d8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:08.194139 10589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 742ff5c87be4406ca1148e7bc873d8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:08.194227 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 742ff5c87be4406ca1148e7bc873d8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:08.201929 10819 tablet_bootstrap.cc:492] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf: Bootstrap starting.
I20260926 20:52:08.202910 10817 tablet_bootstrap.cc:492] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc: Bootstrap starting.
I20260926 20:52:08.203670 10819 tablet_bootstrap.cc:654] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:08.204680 10817 tablet_bootstrap.cc:654] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:08.206509 10819 tablet_bootstrap.cc:492] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf: No bootstrap required, opened a new log
I20260926 20:52:08.206717 10819 ts_tablet_manager.cc:1397] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:52:08.207697 10819 raft_consensus.cc:348] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:08.208024 10819 raft_consensus.cc:374] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:08.208091 10819 raft_consensus.cc:729] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15606209d6c44d17a95c5c4c6008facf, State: Initialized, Role: FOLLOWER
I20260926 20:52:08.208359 10819 consensus_queue.cc:260] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:08.208850 10824 tablet_replica.cc:439] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } }
I20260926 20:52:08.209044 10819 ts_tablet_manager.cc:1428] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:52:08.209157 10824 tablet_replica.cc:442] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:08.210631 10820 tablet_bootstrap.cc:492] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39: Bootstrap starting.
I20260926 20:52:08.212378 10820 tablet_bootstrap.cc:654] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:08.213881 10817 tablet_bootstrap.cc:492] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc: No bootstrap required, opened a new log
I20260926 20:52:08.214092 10817 ts_tablet_manager.cc:1397] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260926 20:52:08.214821 10817 raft_consensus.cc:348] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:08.215040 10817 raft_consensus.cc:374] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:08.215132 10817 raft_consensus.cc:729] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2332a57f502141f289dc5a59fc8b6afc, State: Initialized, Role: FOLLOWER
I20260926 20:52:08.215378 10817 consensus_queue.cc:260] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:08.215788 10823 tablet_replica.cc:439] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } }
I20260926 20:52:08.216143 10823 tablet_replica.cc:442] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:08.216481 10817 ts_tablet_manager.cc:1428] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:52:08.218528 10820 tablet_bootstrap.cc:492] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39: No bootstrap required, opened a new log
I20260926 20:52:08.218751 10820 ts_tablet_manager.cc:1397] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20260926 20:52:08.219597 10820 raft_consensus.cc:348] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:08.219859 10820 raft_consensus.cc:374] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:08.219970 10820 raft_consensus.cc:729] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75eefe06349c4a1785ccbc4be557aa39, State: Initialized, Role: FOLLOWER
I20260926 20:52:08.220227 10820 consensus_queue.cc:260] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:08.220829 10820 ts_tablet_manager.cc:1428] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 20:52:08.220660 10835 tablet_replica.cc:439] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } }
I20260926 20:52:08.221011 10835 tablet_replica.cc:442] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:08.254920 10835 raft_consensus.cc:482] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:08.255162 10835 raft_consensus.cc:504] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:08.255990 10835 leader_election.cc:288] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2332a57f502141f289dc5a59fc8b6afc (127.8.249.129:34463), 15606209d6c44d17a95c5c4c6008facf (127.8.249.130:33505)
I20260926 20:52:08.256558 10472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742ff5c87be4406ca1148e7bc873d8b6" candidate_uuid: "75eefe06349c4a1785ccbc4be557aa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2332a57f502141f289dc5a59fc8b6afc" is_pre_election: true
I20260926 20:52:08.256644 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742ff5c87be4406ca1148e7bc873d8b6" candidate_uuid: "75eefe06349c4a1785ccbc4be557aa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15606209d6c44d17a95c5c4c6008facf" is_pre_election: true
I20260926 20:52:08.256848 10472 raft_consensus.cc:2454] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75eefe06349c4a1785ccbc4be557aa39 in term 0.
I20260926 20:52:08.256953 10609 raft_consensus.cc:2454] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75eefe06349c4a1785ccbc4be557aa39 in term 0.
I20260926 20:52:08.257309 10682 leader_election.cc:302] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2332a57f502141f289dc5a59fc8b6afc, 75eefe06349c4a1785ccbc4be557aa39; no voters:
I20260926 20:52:08.257566 10835 raft_consensus.cc:2781] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:52:08.257687 10835 raft_consensus.cc:482] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:08.257795 10835 raft_consensus.cc:3037] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:08.260113 10835 raft_consensus.cc:504] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:08.260682 10835 leader_election.cc:288] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [CANDIDATE]: Term 1 election: Requested vote from peers 2332a57f502141f289dc5a59fc8b6afc (127.8.249.129:34463), 15606209d6c44d17a95c5c4c6008facf (127.8.249.130:33505)
I20260926 20:52:08.261163 10472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742ff5c87be4406ca1148e7bc873d8b6" candidate_uuid: "75eefe06349c4a1785ccbc4be557aa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2332a57f502141f289dc5a59fc8b6afc"
I20260926 20:52:08.261276 10609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742ff5c87be4406ca1148e7bc873d8b6" candidate_uuid: "75eefe06349c4a1785ccbc4be557aa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15606209d6c44d17a95c5c4c6008facf"
I20260926 20:52:08.261390 10472 raft_consensus.cc:3037] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:08.261538 10609 raft_consensus.cc:3037] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:08.263415 10472 raft_consensus.cc:2454] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75eefe06349c4a1785ccbc4be557aa39 in term 1.
I20260926 20:52:08.263803 10682 leader_election.cc:302] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2332a57f502141f289dc5a59fc8b6afc, 75eefe06349c4a1785ccbc4be557aa39; no voters:
I20260926 20:52:08.264088 10835 raft_consensus.cc:2781] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:08.264248 10835 raft_consensus.cc:686] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [term 1 LEADER]: Becoming Leader. State: Replica: 75eefe06349c4a1785ccbc4be557aa39, State: Running, Role: LEADER
I20260926 20:52:08.264611 10609 raft_consensus.cc:2454] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75eefe06349c4a1785ccbc4be557aa39 in term 1.
I20260926 20:52:08.264804 10835 consensus_queue.cc:237] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } }
I20260926 20:52:08.266564 10827 tablet_replica.cc:439] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39: TxnStatusTablet state changed. Reason: New leader 75eefe06349c4a1785ccbc4be557aa39. Latest consensus state: current_term: 1 leader_uuid: "75eefe06349c4a1785ccbc4be557aa39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } }
I20260926 20:52:08.266996 10827 tablet_replica.cc:442] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:52:08.267697 10860 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:52:08.267933 10860 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:52:08.268608 10313 catalog_manager.cc:5495] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75eefe06349c4a1785ccbc4be557aa39 (127.8.249.131). New cstate: current_term: 1 leader_uuid: "75eefe06349c4a1785ccbc4be557aa39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } health_report { overall_health: HEALTHY } } }
I20260926 20:52:08.299074 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.293921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:41503 (local address 127.0.0.1:44536)
0926 20:52:08.294167 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:08.294188 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:08.294343 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:08.294733 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:08.294737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:08.294750 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:52:08.294954 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:08.294963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.295978 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.295986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:08.297097 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:08.297108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.297837 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.297843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:08.298007 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.298093 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:52:08.298500 (+ 407us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:52:08.298523 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:52:08.298608 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:52:08.298762 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 20:52:08.299075 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.293993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.8.249.131:41503)
0926 20:52:08.294186 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:08.294192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:08.294256 (+ 64us) server_negotiation.cc:407] Connection header received
0926 20:52:08.294533 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:08.294537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:08.294609 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:08.294694 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:08.295109 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.295850 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.297235 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.297693 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.297822 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.298289 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:08.298394 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:08.298492 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:08.298639 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 20:52:08.298737 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:52:08.302618 10472 raft_consensus.cc:1261] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc [term 1 FOLLOWER]: Refusing update from remote peer 75eefe06349c4a1785ccbc4be557aa39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:08.302664 10609 raft_consensus.cc:1261] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf [term 1 FOLLOWER]: Refusing update from remote peer 75eefe06349c4a1785ccbc4be557aa39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:08.303304 10835 consensus_queue.cc:1035] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:08.303586 10827 consensus_queue.cc:1035] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:08.312579 10823 tablet_replica.cc:439] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc: TxnStatusTablet state changed. Reason: New leader 75eefe06349c4a1785ccbc4be557aa39. Latest consensus state: current_term: 1 leader_uuid: "75eefe06349c4a1785ccbc4be557aa39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } }
I20260926 20:52:08.312896 10823 tablet_replica.cc:442] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:08.313355 10824 tablet_replica.cc:439] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf: TxnStatusTablet state changed. Reason: New leader 75eefe06349c4a1785ccbc4be557aa39. Latest consensus state: current_term: 1 leader_uuid: "75eefe06349c4a1785ccbc4be557aa39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } }
I20260926 20:52:08.313659 10824 tablet_replica.cc:442] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:08.315249 10830 mvcc.cc:205] Tried to move back new op lower bound from 7333707482321682432 to 7333707482174939136. Current Snapshot: MvccSnapshot[applied={T|T < 7333707482321682432}]
I20260926 20:52:08.317448 10827 tablet_replica.cc:439] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75eefe06349c4a1785ccbc4be557aa39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } }
I20260926 20:52:08.317720 10827 tablet_replica.cc:442] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:52:08.318492 10835 tablet_replica.cc:439] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75eefe06349c4a1785ccbc4be557aa39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } }
I20260926 20:52:08.318713 10835 tablet_replica.cc:442] T 742ff5c87be4406ca1148e7bc873d8b6 P 75eefe06349c4a1785ccbc4be557aa39: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:52:08.318852 10823 tablet_replica.cc:439] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75eefe06349c4a1785ccbc4be557aa39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } }
I20260926 20:52:08.319051 10823 tablet_replica.cc:442] T 742ff5c87be4406ca1148e7bc873d8b6 P 2332a57f502141f289dc5a59fc8b6afc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:08.320704 10824 tablet_replica.cc:439] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75eefe06349c4a1785ccbc4be557aa39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 } } peers { permanent_uuid: "15606209d6c44d17a95c5c4c6008facf" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 33505 } } peers { permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 } } }
I20260926 20:52:08.320988 10824 tablet_replica.cc:442] T 742ff5c87be4406ca1148e7bc873d8b6 P 15606209d6c44d17a95c5c4c6008facf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:08.321386 10832 mvcc.cc:205] Tried to move back new op lower bound from 7333707482321682432 to 7333707482174939136. Current Snapshot: MvccSnapshot[applied={T|T < 7333707482321682432 or (T in {7333707482321682432})}]
I20260926 20:52:08.334118 10864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.327500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.8.249.130:33505)
0926 20:52:08.327993 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:08.327997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:08.328112 (+ 115us) server_negotiation.cc:407] Connection header received
0926 20:52:08.328357 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:08.328361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:08.328437 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:08.328574 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:08.329033 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.329773 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.331868 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.332274 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.332366 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.332943 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:08.333488 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:08.333591 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:08.333744 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 20:52:08.333808 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":274,"threads_started":1}
I20260926 20:52:08.350484 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.343972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:39097 (local address 127.8.249.131:41503)
0926 20:52:08.344215 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:08.344225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:08.344666 (+ 441us) server_negotiation.cc:407] Connection header received
0926 20:52:08.344915 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:08.344918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:08.344987 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:08.345060 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:08.345775 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.346687 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.348756 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.349242 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.349348 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.349846 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:08.349972 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:08.350044 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:08.350180 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 20:52:08.350243 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:52:08.350672 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.343926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:41503 (local address 127.8.249.130:39097)
0926 20:52:08.344559 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:08.344581 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:08.344746 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:08.345114 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:08.345143 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:08.345158 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:52:08.345514 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:08.345527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.346855 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.346862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:08.348601 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:08.348615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.349423 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.349431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:08.349606 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.349678 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:52:08.350066 (+ 388us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:52:08.350078 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 20:52:08.350156 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:52:08.350291 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":317,"threads_started":1}
I20260926 20:52:08.361073 10472 raft_consensus.cc:1261] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc [term 1 FOLLOWER]: Refusing update from remote peer 15606209d6c44d17a95c5c4c6008facf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:08.361214 10748 raft_consensus.cc:1261] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39 [term 1 FOLLOWER]: Refusing update from remote peer 15606209d6c44d17a95c5c4c6008facf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:08.361905 10848 consensus_queue.cc:1035] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [LEADER]: Connected to new peer: Peer: permanent_uuid: "75eefe06349c4a1785ccbc4be557aa39" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:08.362289 10824 consensus_queue.cc:1035] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2332a57f502141f289dc5a59fc8b6afc" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 34463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:08.400616 10864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.394193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:50273 (local address 127.8.249.130:33505)
0926 20:52:08.394621 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:08.394647 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:08.395066 (+ 419us) server_negotiation.cc:407] Connection header received
0926 20:52:08.395363 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:08.395369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:08.395444 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:08.395535 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:08.396068 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.396900 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.398549 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.399190 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.399281 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.399911 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:08.400052 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:08.400150 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:08.400350 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 20:52:08.400421 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260926 20:52:08.400945 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.394274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:33505 (local address 127.8.249.131:50273)
0926 20:52:08.394970 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:08.394990 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:08.395172 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:08.395563 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:08.395568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:08.395582 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:52:08.395909 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:08.395917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.397031 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.397036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:08.398367 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:08.398386 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.399481 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.399489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:08.399668 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.399741 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:52:08.400209 (+ 468us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:52:08.400224 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 20:52:08.400311 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:52:08.400485 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":585,"threads_started":1}
I20260926 20:52:08.448740 10313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:08.449021 10313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:08.452587 10313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec41f31db26146acb61d64807902ecce: Sending DeleteTablet for 3 replicas of tablet 9e36c73480544f069b52c6e63354bf27
I20260926 20:52:08.453531 10589 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e36c73480544f069b52c6e63354bf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:08 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:37366
I20260926 20:52:08.453603 10728 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e36c73480544f069b52c6e63354bf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:08 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:44512
I20260926 20:52:08.453814 10840 tablet_replica.cc:330] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf: stopping tablet replica
I20260926 20:52:08.454028 10840 raft_consensus.cc:2229] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:52:08.454226 10450 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e36c73480544f069b52c6e63354bf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:08 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:44546
I20260926 20:52:08.454617 10840 raft_consensus.cc:2258] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:08.456173 10840 ts_tablet_manager.cc:1905] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:08.467240 10878 tablet_replica.cc:330] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc: stopping tablet replica
I20260926 20:52:08.467499 10878 raft_consensus.cc:2229] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:08.467774 10878 raft_consensus.cc:2258] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:08.469161 10878 ts_tablet_manager.cc:1905] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:08.474861 10840 ts_tablet_manager.cc:1918] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:52:08.475121 10840 log.cc:1196] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/9e36c73480544f069b52c6e63354bf27
I20260926 20:52:08.475603 10840 ts_tablet_manager.cc:1939] T 9e36c73480544f069b52c6e63354bf27 P 15606209d6c44d17a95c5c4c6008facf: Deleting consensus metadata
I20260926 20:52:08.475888 10877 tablet_replica.cc:330] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39: stopping tablet replica
I20260926 20:52:08.476104 10877 raft_consensus.cc:2229] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:08.476336 10877 raft_consensus.cc:2258] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:08.476941 10300 catalog_manager.cc:4841] TS 15606209d6c44d17a95c5c4c6008facf (127.8.249.130:33505): tablet 9e36c73480544f069b52c6e63354bf27 (table test-table [id=a679bd8dbfcc48b5b51c0b12cdea46bd]) successfully deleted
I20260926 20:52:08.477749 10878 ts_tablet_manager.cc:1918] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:52:08.477880 10878 log.cc:1196] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/9e36c73480544f069b52c6e63354bf27
I20260926 20:52:08.478314 10878 ts_tablet_manager.cc:1939] T 9e36c73480544f069b52c6e63354bf27 P 2332a57f502141f289dc5a59fc8b6afc: Deleting consensus metadata
I20260926 20:52:08.478626 10877 ts_tablet_manager.cc:1905] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:08.485950 10877 ts_tablet_manager.cc:1918] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:52:08.486102 10877 log.cc:1196] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/9e36c73480544f069b52c6e63354bf27
I20260926 20:52:08.486541 10877 ts_tablet_manager.cc:1939] T 9e36c73480544f069b52c6e63354bf27 P 75eefe06349c4a1785ccbc4be557aa39: Deleting consensus metadata
I20260926 20:52:08.487805 10299 catalog_manager.cc:4841] TS 75eefe06349c4a1785ccbc4be557aa39 (127.8.249.131:41503): tablet 9e36c73480544f069b52c6e63354bf27 (table test-table [id=a679bd8dbfcc48b5b51c0b12cdea46bd]) successfully deleted
I20260926 20:52:08.491511 10300 catalog_manager.cc:4841] TS 2332a57f502141f289dc5a59fc8b6afc (127.8.249.129:34463): tablet 9e36c73480544f069b52c6e63354bf27 (table test-table [id=a679bd8dbfcc48b5b51c0b12cdea46bd]) successfully deleted
Sep 26 20:52:08 dist-test-slave-2dg3 krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455927, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:52:08.509210 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.477668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.8.249.129:34463)
0926 20:52:08.477856 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:08.477863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:08.488723 (+ 10860us) server_negotiation.cc:407] Connection header received
0926 20:52:08.488908 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:08.488915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:08.489039 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:08.489154 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:08.491146 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.492308 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.494254 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.494980 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.495103 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.501000 (+ 5897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:08.501022 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:08.501044 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:08.501101 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:08.505244 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:08.506306 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:08.506319 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:08.506324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:08.506528 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:08.507086 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:08.507092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:08.507095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:08.507566 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:08.507894 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:08.508531 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 20:52:08.508806 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 20:52:08.509919 10515 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44554
I20260926 20:52:08.530340 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.513174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.8.249.190:35217)
0926 20:52:08.513385 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:08.513392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:08.514220 (+ 828us) server_negotiation.cc:407] Connection header received
0926 20:52:08.514387 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:08.514393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:08.514479 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:08.514585 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:08.516331 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.517484 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.519351 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.520030 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.520170 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.523332 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:08.523340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:08.523344 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:08.523392 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:08.527047 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:08.528106 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:08.528112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:08.528116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:08.528263 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:08.528691 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:08.528696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:08.528700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:08.529108 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:08.529420 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:08.529726 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 20:52:08.529967 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 20:52:08.531077 10313 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50944
I20260926 20:52:08.533829 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 10384
I20260926 20:52:08.558342 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.535104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.8.249.129:34463)
0926 20:52:08.535298 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:08.535305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:08.539347 (+ 4042us) server_negotiation.cc:407] Connection header received
0926 20:52:08.539507 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:08.539513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:08.539602 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:08.539711 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:08.541506 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.542662 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.544538 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.545219 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.545356 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.548586 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:08.548594 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:08.548599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:08.548647 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:08.552171 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:08.553960 (+ 1789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:08.553968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:08.553975 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:08.554219 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:08.555127 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:08.555132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:08.555163 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:08.555672 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:08.556056 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:08.557138 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0926 20:52:08.557480 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 20:52:08.561510 10515 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44556
W20260926 20:52:08.562712 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 10384: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:52:08.562878 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 10384
I20260926 20:52:08.587723 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 10524
Sep 26 20:52:08 dist-test-slave-2dg3 krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455927, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:52:08.629454 10864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.588546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.8.249.130:33505)
0926 20:52:08.588793 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:08.588799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:08.589076 (+ 277us) server_negotiation.cc:407] Connection header received
0926 20:52:08.589150 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:08.589156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:08.589265 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:08.589365 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:08.591619 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.592714 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.594707 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.595490 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.595620 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.610068 (+ 14448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:08.610092 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:08.610113 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:08.610175 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:08.614495 (+ 4320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:08.615698 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:08.615709 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:08.615713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:08.615890 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:08.616369 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:08.616374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:08.616378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:08.626868 (+ 10490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:08.627292 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:08.628833 (+ 1541us) server_negotiation.cc:299] Negotiation successful
0926 20:52:08.629077 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 20:52:08.630191 10654 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37376
W20260926 20:52:08.630977 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 10524: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:52:08.631183 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 10524
I20260926 20:52:08.650669 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 10663
Sep 26 20:52:08 dist-test-slave-2dg3 krb5kdc[10268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455927, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:52:08.666642 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:08.651742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.8.249.131:41503)
0926 20:52:08.651928 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:08.651935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:08.651981 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:52:08.652184 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:08.652189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:08.652262 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:08.652369 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:08.653617 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.654401 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.655697 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:08.656190 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:08.656334 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:08.660578 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:08.660591 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:08.660607 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:08.660643 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:08.663823 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:08.664538 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:08.664546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:08.664548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:08.664665 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:08.665025 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:08.665032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:08.665038 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:08.665422 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:08.665785 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:08.666009 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 20:52:08.666240 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 20:52:08.667423 10793 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44544
W20260926 20:52:08.668114 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 10663: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:52:08.668268 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 10663
I20260926 20:52:08.690433 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 10281
I20260926 20:52:08.691659 10353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:08.805842 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 10281
2026-09-26T20:52:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4545 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:52:08 dist-test-slave-2dg3 krb5kdc[10887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:52:08 dist-test-slave-2dg3 krb5kdc[10887](info): set up 2 sockets
Sep 26 20:52:08 dist-test-slave-2dg3 krb5kdc[10887](info): commencing operation
krb5kdc: starting...
W20260926 20:52:09.883248 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:52:10 dist-test-slave-2dg3 krb5kdc[10887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:52:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:52:10.070561 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:42267
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:41819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:42267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:10.393066 10900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:10.393353 10900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:10.393505 10900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:10.400722 10900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:10.400825 10900 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:52:10.400897 10900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:10.401000 10900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:10.401070 10900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:10.401165 10900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:10.410811 10900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:41819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:42267
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:42267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:10.412032 10900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:10.413319 10900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:10.425288 10908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:10.425415 10905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:10.425333 10906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:10.425992 10900 server_base.cc:1034] running on GCE node
I20260926 20:52:10.426728 10900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:10.428102 10900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:10.429342 10900 hybrid_clock.cc:648] HybridClock initialized: now 1790455930429286 us; error 25 us; skew 500 ppm
I20260926 20:52:10.429682 10900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:10 dist-test-slave-2dg3 krb5kdc[10887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455930, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:10.432722 10900 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:52:10.434171 10900 webserver.cc:460] Webserver started at http://127.8.249.190:34181/ using document root <none> and password file <none>
I20260926 20:52:10.434931 10900 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:10.435052 10900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:10.435365 10900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:10.438287 10900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "ab697a91e2124def80684f69c02e7fc4"
format_stamp: "Formatted at 2026-09-26 20:52:10 on dist-test-slave-2dg3"
server_key: "27056b54a9349357517c6fd0ba77c49a"
server_key_iv: "ed7ee61db1ebd54f8832cca207544ad9"
server_key_version: "encryptionkey@0"
I20260926 20:52:10.439100 10900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "ab697a91e2124def80684f69c02e7fc4"
format_stamp: "Formatted at 2026-09-26 20:52:10 on dist-test-slave-2dg3"
server_key: "27056b54a9349357517c6fd0ba77c49a"
server_key_iv: "ed7ee61db1ebd54f8832cca207544ad9"
server_key_version: "encryptionkey@0"
I20260926 20:52:10.444248 10900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:52:10.447697 10915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:10.449014 10900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:52:10.449190 10900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "ab697a91e2124def80684f69c02e7fc4"
format_stamp: "Formatted at 2026-09-26 20:52:10 on dist-test-slave-2dg3"
server_key: "27056b54a9349357517c6fd0ba77c49a"
server_key_iv: "ed7ee61db1ebd54f8832cca207544ad9"
server_key_version: "encryptionkey@0"
I20260926 20:52:10.449411 10900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:10.464294 10900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:10.468748 10900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:10.469026 10900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:10.489924 10900 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:42267
I20260926 20:52:10.489948 10976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:42267 every 8 connection(s)
I20260926 20:52:10.491370 10900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:10.495142 10977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:10.498737 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 10900
I20260926 20:52:10.498983 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:10.499779 9190 external_mini_cluster.cc:1428] Setting key 0d2f417e831eb97d7b5645fa905deeb0
I20260926 20:52:10.508464 10977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4: Bootstrap starting.
I20260926 20:52:10.512178 10977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:10.513484 10977 log.cc:824] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:52:10 dist-test-slave-2dg3 krb5kdc[10887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:10.516577 10977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4: No bootstrap required, opened a new log
I20260926 20:52:10.523236 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:10.501882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.8.249.190:42267)
0926 20:52:10.502661 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:10.502672 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:10.502714 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:52:10.504417 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:10.504440 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:10.504863 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:10.505213 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:10.506972 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:10.508250 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:10.509895 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:10.510577 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:10.510722 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:10.515803 (+ 5081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:10.515810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:10.515824 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:10.515861 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:10.518994 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:10.519778 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:10.519785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:10.519791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:10.519959 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:10.520353 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:10.520356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:10.520358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:10.520868 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:10.521128 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:10.521550 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 20:52:10.521955 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":393,"threads_started":1}
I20260926 20:52:10.525589 10977 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab697a91e2124def80684f69c02e7fc4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42267 } }
I20260926 20:52:10.525949 10977 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:10.526069 10977 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab697a91e2124def80684f69c02e7fc4, State: Initialized, Role: FOLLOWER
I20260926 20:52:10.526657 10977 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab697a91e2124def80684f69c02e7fc4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42267 } }
I20260926 20:52:10.526928 10977 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:10.527072 10977 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:10.527240 10977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:10.529316 10977 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab697a91e2124def80684f69c02e7fc4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42267 } }
I20260926 20:52:10.529780 10977 leader_election.cc:302] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab697a91e2124def80684f69c02e7fc4; no voters:
I20260926 20:52:10.530418 10977 leader_election.cc:288] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:10.530583 10982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:10.531265 10982 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [term 1 LEADER]: Becoming Leader. State: Replica: ab697a91e2124def80684f69c02e7fc4, State: Running, Role: LEADER
I20260926 20:52:10.531793 10982 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab697a91e2124def80684f69c02e7fc4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42267 } }
I20260926 20:52:10.532617 10977 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:10.534940 10984 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab697a91e2124def80684f69c02e7fc4. Latest consensus state: current_term: 1 leader_uuid: "ab697a91e2124def80684f69c02e7fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab697a91e2124def80684f69c02e7fc4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42267 } } }
I20260926 20:52:10.534565 10983 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab697a91e2124def80684f69c02e7fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab697a91e2124def80684f69c02e7fc4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42267 } } }
I20260926 20:52:10.535358 10983 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:10.535367 10984 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:10.537954 10990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:10.543321 10990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:10.552408 10990 catalog_manager.cc:1347] Generated new cluster ID: 2e1ea2cabf7c44a7981eab8f50aff2c2
I20260926 20:52:10.552525 10990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:10.561636 10990 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:10.562975 10990 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:10.575378 10990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4: Generated new TSK 0
I20260926 20:52:10.576241 10990 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:52:10.625770 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:42267
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:41819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:10.972041 11003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:10.972347 11003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:10.972541 11003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:10.980041 11003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:10.980204 11003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:10.980382 11003 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:52:10.980470 11003 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:52:10.980609 11003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:52:10.990697 11003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:41819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.249.190:42267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:10.992096 11003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:10.993499 11003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:11.007529 11011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:11.007560 11008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:11.007529 11009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:11.007982 11003 server_base.cc:1034] running on GCE node
I20260926 20:52:11.008766 11003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:11.009701 11003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:11.010910 11003 hybrid_clock.cc:648] HybridClock initialized: now 1790455931010856 us; error 40 us; skew 500 ppm
I20260926 20:52:11.011231 11003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:11 dist-test-slave-2dg3 krb5kdc[10887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455931, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:11.014401 11003 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:52:11.016024 11003 webserver.cc:460] Webserver started at http://127.8.249.129:38143/ using document root <none> and password file <none>
I20260926 20:52:11.016752 11003 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:11.016896 11003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:11.017210 11003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:11.020342 11003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "2bbe00975fc74ad6b99f2b4140611cfe"
format_stamp: "Formatted at 2026-09-26 20:52:11 on dist-test-slave-2dg3"
server_key: "56674a02b46a6d6685497b5d1f8c832b"
server_key_iv: "860efb5ec93188342b1a754aec912e40"
server_key_version: "encryptionkey@0"
I20260926 20:52:11.021121 11003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "2bbe00975fc74ad6b99f2b4140611cfe"
format_stamp: "Formatted at 2026-09-26 20:52:11 on dist-test-slave-2dg3"
server_key: "56674a02b46a6d6685497b5d1f8c832b"
server_key_iv: "860efb5ec93188342b1a754aec912e40"
server_key_version: "encryptionkey@0"
I20260926 20:52:11.026400 11003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:52:11.029874 11018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:11.031275 11003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:11.031453 11003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "2bbe00975fc74ad6b99f2b4140611cfe"
format_stamp: "Formatted at 2026-09-26 20:52:11 on dist-test-slave-2dg3"
server_key: "56674a02b46a6d6685497b5d1f8c832b"
server_key_iv: "860efb5ec93188342b1a754aec912e40"
server_key_version: "encryptionkey@0"
I20260926 20:52:11.031652 11003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:11.053889 11003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:11.058334 11003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:11.058640 11003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:11.061205 11003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:11.061343 11003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:11.061466 11003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:11.061546 11003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:52:11 dist-test-slave-2dg3 krb5kdc[10887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455931, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:11.084785 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:11.061996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:44777 (local address 127.8.249.190:42267)
0926 20:52:11.062414 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:11.062418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:11.063777 (+ 1359us) server_negotiation.cc:407] Connection header received
0926 20:52:11.065805 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:11.065811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:11.065896 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:11.066017 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:11.068243 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.069552 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.071946 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.072584 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.072740 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:11.077676 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:11.077681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:11.077684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:11.077715 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:11.080716 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:11.081638 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:11.081644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:11.081647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:11.081798 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:11.082368 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:11.082373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:11.082377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:11.082808 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:11.082951 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:11.083938 (+ 987us) server_negotiation.cc:299] Negotiation successful
0926 20:52:11.084241 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":219,"threads_started":1}
I20260926 20:52:11.085424 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:11.062770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42267 (local address 127.8.249.129:44777)
0926 20:52:11.063626 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:11.063670 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:11.065538 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:11.066299 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:11.066314 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:11.067157 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:11.067968 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:11.068016 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.069726 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.069737 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:11.071787 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:11.071799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.073062 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.073067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:11.073329 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:11.074253 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:11.074296 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:11.077427 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:11.080846 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:11.080855 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:11.080870 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:11.081484 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:11.081949 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:11.081956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:11.081962 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:11.082241 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:11.082919 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:11.082926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:11.083488 (+ 562us) client_negotiation.cc:769] Sending connection context
0926 20:52:11.083813 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 20:52:11.084270 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":351,"threads_started":1}
I20260926 20:52:11.107659 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:11.091619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:57785 (local address 127.8.249.190:42267)
0926 20:52:11.091854 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:11.091863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:11.092527 (+ 664us) server_negotiation.cc:407] Connection header received
0926 20:52:11.092771 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:11.092776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:11.092848 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:11.092929 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:11.094547 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.095424 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.097469 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.097884 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.098020 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:11.100390 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:11.100400 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:11.100408 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:11.100462 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:11.104208 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:11.105409 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:11.105420 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:11.105426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:11.105616 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:11.106000 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:11.106009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:11.106014 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:11.106466 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:11.106645 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:11.107066 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 20:52:11.107282 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 20:52:11.107758 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:11.091589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42267 (local address 127.8.249.129:57785)
0926 20:52:11.092384 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:11.092410 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:11.092619 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:11.093004 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:11.093012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:11.093844 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:11.094376 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:11.094395 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.095570 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.095578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:11.097330 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:11.097342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.098077 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.098085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:11.098254 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:11.099074 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:11.099104 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:11.100163 (+ 1059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:11.104355 (+ 4192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:11.104361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:11.104370 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:11.105181 (+ 811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:11.105742 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:11.105745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:11.105747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:11.105887 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:11.106589 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:11.106592 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:11.106745 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 20:52:11.106952 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 20:52:11.107301 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":380,"threads_started":1}
I20260926 20:52:11.117611 11003 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:42499
I20260926 20:52:11.117642 11138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:42499 every 8 connection(s)
I20260926 20:52:11.119097 11003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:52:11.121215 11139 heartbeater.cc:347] Connected to a master server at 127.8.249.190:42267
I20260926 20:52:11.121436 11139 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:11.122051 11139 heartbeater.cc:511] Master 127.8.249.190:42267 requested a full tablet report, sending...
I20260926 20:52:11.123935 10932 ts_manager.cc:194] Registered new tserver with Master: 2bbe00975fc74ad6b99f2b4140611cfe (127.8.249.129:42499)
I20260926 20:52:11.125357 10932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:44777
I20260926 20:52:11.126931 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 11003
I20260926 20:52:11.127123 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:52:11.127733 9190 external_mini_cluster.cc:1428] Setting key 7c4d60289e40474caf63517735a6a901
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:52:11.167415 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:42267
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:41819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:11.489771 11143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:11.490224 11143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:11.490412 11143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:11.502043 11143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:11.502187 11143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:11.502312 11143 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:52:11.502384 11143 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:52:11.502481 11143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:52:11.516062 11143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:41819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.249.190:42267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:11.517302 11143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:11.518560 11143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:11.532225 11148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:11.532369 11149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:11.532249 11151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:11.533347 11143 server_base.cc:1034] running on GCE node
I20260926 20:52:11.533965 11143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:11.534945 11143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:11.536132 11143 hybrid_clock.cc:648] HybridClock initialized: now 1790455931536089 us; error 37 us; skew 500 ppm
I20260926 20:52:11.536458 11143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:11 dist-test-slave-2dg3 krb5kdc[10887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455931, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:11.539530 11143 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:52:11.541025 11143 webserver.cc:460] Webserver started at http://127.8.249.130:38781/ using document root <none> and password file <none>
I20260926 20:52:11.541704 11143 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:11.541836 11143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:11.542132 11143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:11.545011 11143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "94682c641b8b457cbe5552c065e73a32"
format_stamp: "Formatted at 2026-09-26 20:52:11 on dist-test-slave-2dg3"
server_key: "2ce86a6e358ef58554ee41107f2f1a32"
server_key_iv: "ba8eff52dbe06693f7f0350e5a305cc2"
server_key_version: "encryptionkey@0"
I20260926 20:52:11.545866 11143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "94682c641b8b457cbe5552c065e73a32"
format_stamp: "Formatted at 2026-09-26 20:52:11 on dist-test-slave-2dg3"
server_key: "2ce86a6e358ef58554ee41107f2f1a32"
server_key_iv: "ba8eff52dbe06693f7f0350e5a305cc2"
server_key_version: "encryptionkey@0"
I20260926 20:52:11.551007 11143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 20:52:11.554456 11158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:11.556003 11143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:11.556196 11143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "94682c641b8b457cbe5552c065e73a32"
format_stamp: "Formatted at 2026-09-26 20:52:11 on dist-test-slave-2dg3"
server_key: "2ce86a6e358ef58554ee41107f2f1a32"
server_key_iv: "ba8eff52dbe06693f7f0350e5a305cc2"
server_key_version: "encryptionkey@0"
I20260926 20:52:11.556411 11143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:11.580109 11143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:11.584610 11143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:11.584903 11143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:11.587899 11143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:11.588054 11143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:11.588181 11143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:11.588279 11143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:52:11 dist-test-slave-2dg3 krb5kdc[10887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455931, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:11.615468 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:11.588533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:42821 (local address 127.8.249.190:42267)
0926 20:52:11.588738 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:11.588744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:11.590221 (+ 1477us) server_negotiation.cc:407] Connection header received
0926 20:52:11.592356 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:11.592362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:11.592477 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:11.592622 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:11.595141 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.596479 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.598924 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.599680 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.599809 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:11.605502 (+ 5693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:11.605511 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:11.605516 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:11.605565 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:11.610544 (+ 4979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:11.611918 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:11.611927 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:11.611933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:11.612097 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:11.613134 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:11.613142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:11.613147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:11.613504 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:11.613697 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:11.614874 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0926 20:52:11.615113 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:52:11.616654 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:11.589113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42267 (local address 127.8.249.130:42821)
0926 20:52:11.590036 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:11.590084 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:11.592000 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:11.593032 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:11.593040 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:11.593865 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:11.594865 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:11.594889 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.596656 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.596664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:11.598731 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:11.598747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.600016 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.600060 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:11.600369 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:11.601302 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:11.601360 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:11.605155 (+ 3795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:11.610742 (+ 5587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:11.610753 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:11.610782 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:11.611714 (+ 932us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:11.612234 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:11.612240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:11.612244 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:11.612598 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:11.613797 (+ 1199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:11.613809 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:11.614358 (+ 549us) client_negotiation.cc:769] Sending connection context
0926 20:52:11.614721 (+ 363us) client_negotiation.cc:240] Negotiation successful
0926 20:52:11.615359 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":390,"threads_started":1}
I20260926 20:52:11.637338 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:11.622722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:35817 (local address 127.8.249.190:42267)
0926 20:52:11.622938 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:11.622943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:11.623774 (+ 831us) server_negotiation.cc:407] Connection header received
0926 20:52:11.623998 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:11.624008 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:11.624071 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:11.624171 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:11.625116 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.625822 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.627332 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.627767 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.627912 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:11.630685 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:11.630690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:11.630694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:11.630725 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:11.634143 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:11.635173 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:11.635178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:11.635181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:11.635363 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:11.635880 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:11.635888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:11.635891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:11.636276 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:11.636406 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:11.636943 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 20:52:11.637112 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 20:52:11.637916 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:11.623089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42267 (local address 127.8.249.130:35817)
0926 20:52:11.623663 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:11.623679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:11.623871 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:11.624193 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:11.624198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:11.624686 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:11.624968 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:11.624976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.625954 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.625960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:11.627185 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:11.627202 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:11.627953 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:11.627959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:11.628178 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:11.628954 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:11.628991 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:11.630483 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:11.634269 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:11.634277 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:11.634281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:11.634938 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:11.635512 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:11.635537 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:11.635544 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:11.635733 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:11.636396 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:11.636404 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:11.636670 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 20:52:11.636846 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 20:52:11.637274 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":317,"threads_started":1}
I20260926 20:52:11.648157 11143 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:36381
I20260926 20:52:11.648196 11277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:36381 every 8 connection(s)
I20260926 20:52:11.649438 11143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:52:11.651527 11278 heartbeater.cc:347] Connected to a master server at 127.8.249.190:42267
I20260926 20:52:11.651751 11278 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:11.652376 11278 heartbeater.cc:511] Master 127.8.249.190:42267 requested a full tablet report, sending...
I20260926 20:52:11.653766 10932 ts_manager.cc:194] Registered new tserver with Master: 94682c641b8b457cbe5552c065e73a32 (127.8.249.130:36381)
I20260926 20:52:11.654690 10932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:42821
I20260926 20:52:11.657370 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 11143
I20260926 20:52:11.657544 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:52:11.657972 9190 external_mini_cluster.cc:1428] Setting key 06c240441fa4dfaf7ec46b3a55053018
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:52:11.698427 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:42267
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:41819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:12.030118 11282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:12.030411 11282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:12.030512 11282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:12.038394 11282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:12.038525 11282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:12.038622 11282 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:52:12.038679 11282 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:52:12.038748 11282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:52:12.049518 11282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:41819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.249.190:42267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:12.050858 11282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:12.052201 11282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:12.065438 11287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:12.065407 11288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:12.065887 11290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:12.067091 11282 server_base.cc:1034] running on GCE node
I20260926 20:52:12.067739 11282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:12.068679 11282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:12.069921 11282 hybrid_clock.cc:648] HybridClock initialized: now 1790455932069840 us; error 70 us; skew 500 ppm
I20260926 20:52:12.070237 11282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:12 dist-test-slave-2dg3 krb5kdc[10887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455932, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:12.076851 11282 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:52:12.078403 11282 webserver.cc:460] Webserver started at http://127.8.249.131:33207/ using document root <none> and password file <none>
I20260926 20:52:12.079185 11282 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:12.079291 11282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:12.079553 11282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:12.082435 11282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "99df0a67d849469fa222ca16b8034ffc"
format_stamp: "Formatted at 2026-09-26 20:52:12 on dist-test-slave-2dg3"
server_key: "e4ea81b94cea606c50b268f97f8c80f5"
server_key_iv: "85cccae0a8e650c5c5af2ec79337b11f"
server_key_version: "encryptionkey@0"
I20260926 20:52:12.083235 11282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "99df0a67d849469fa222ca16b8034ffc"
format_stamp: "Formatted at 2026-09-26 20:52:12 on dist-test-slave-2dg3"
server_key: "e4ea81b94cea606c50b268f97f8c80f5"
server_key_iv: "85cccae0a8e650c5c5af2ec79337b11f"
server_key_version: "encryptionkey@0"
I20260926 20:52:12.088410 11282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 20:52:12.091745 11297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:12.093051 11282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:52:12.093178 11282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "99df0a67d849469fa222ca16b8034ffc"
format_stamp: "Formatted at 2026-09-26 20:52:12 on dist-test-slave-2dg3"
server_key: "e4ea81b94cea606c50b268f97f8c80f5"
server_key_iv: "85cccae0a8e650c5c5af2ec79337b11f"
server_key_version: "encryptionkey@0"
I20260926 20:52:12.093344 11282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:12.109560 11282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:12.113262 11282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:12.113494 11282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:12.116110 11282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:12.116201 11282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:12.116417 11282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:12.116520 11282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:12.128233 11139 heartbeater.cc:503] Master 127.8.249.190:42267 was elected leader, sending a full tablet report...
Sep 26 20:52:12 dist-test-slave-2dg3 krb5kdc[10887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455932, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:12.141942 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.116614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:39475 (local address 127.8.249.190:42267)
0926 20:52:12.116820 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.116826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.118639 (+ 1813us) server_negotiation.cc:407] Connection header received
0926 20:52:12.122123 (+ 3484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.122131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.122253 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.122336 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:12.125022 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.126405 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.129168 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.129914 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.130068 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.135559 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:12.135564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:12.135568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:12.135599 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:12.138452 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:12.139600 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:12.139608 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:12.139611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:12.139752 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:12.140232 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:12.140237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:12.140241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:12.140516 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:12.140673 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.141385 (+ 712us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.141607 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:52:12.143389 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.117199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42267 (local address 127.8.249.131:39475)
0926 20:52:12.118448 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:12.118503 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:12.121685 (+ 3182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:12.123033 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:12.123044 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:12.123883 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:12.124761 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.124800 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.126612 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.126620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.128932 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.128948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.130223 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.130230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.130412 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.131249 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:12.131290 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:12.135291 (+ 4001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:12.138597 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:12.138610 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:12.138633 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:12.139441 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:12.139889 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:12.139896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:12.139901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:12.140097 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:12.140646 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:12.140653 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:12.140994 (+ 341us) client_negotiation.cc:769] Sending connection context
0926 20:52:12.141248 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 20:52:12.141788 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":634,"threads_started":1}
I20260926 20:52:12.166695 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.151010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:58083 (local address 127.8.249.190:42267)
0926 20:52:12.151215 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.151223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.152054 (+ 831us) server_negotiation.cc:407] Connection header received
0926 20:52:12.152259 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.152262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.152340 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.152429 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:12.153996 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.155078 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.157609 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.158111 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.158263 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.161109 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:12.161114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:12.161117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:12.161150 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:12.164146 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:12.164787 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:12.164794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:12.164798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:12.164923 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:12.165286 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:12.165291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:12.165294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:12.165566 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:12.165727 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.166100 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.166307 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:52:12.166852 11361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.151067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42267 (local address 127.8.249.131:58083)
0926 20:52:12.151903 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:12.151931 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:12.152129 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:12.152499 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:12.152507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:12.153275 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:12.153802 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.153820 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.155244 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.155252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.157408 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.157429 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.158333 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.158348 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.158634 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.159583 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:12.159600 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:12.160897 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:12.164278 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:12.164283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:12.164288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:12.164669 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:12.165038 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:12.165043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:12.165047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:12.165175 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:12.165680 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:12.165685 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:12.165868 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 20:52:12.166000 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 20:52:12.166323 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":443,"threads_started":1}
I20260926 20:52:12.176080 11282 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:46559
I20260926 20:52:12.176122 11416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:46559 every 8 connection(s)
I20260926 20:52:12.177377 11282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:52:12.179354 11417 heartbeater.cc:347] Connected to a master server at 127.8.249.190:42267
I20260926 20:52:12.179579 11417 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:12.180213 11417 heartbeater.cc:511] Master 127.8.249.190:42267 requested a full tablet report, sending...
I20260926 20:52:12.181439 10932 ts_manager.cc:194] Registered new tserver with Master: 99df0a67d849469fa222ca16b8034ffc (127.8.249.131:46559)
I20260926 20:52:12.182237 10932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:39475
I20260926 20:52:12.187083 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 11282
I20260926 20:52:12.187217 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:52:12.187618 9190 external_mini_cluster.cc:1428] Setting key cec0ab9366c04a467a9842d355a6aadf
I20260926 20:52:12.190598 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:52:12 dist-test-slave-2dg3 krb5kdc[10887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:52:12 dist-test-slave-2dg3 krb5kdc[10887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:12.244891 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.228823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49598 (local address 127.8.249.190:42267)
0926 20:52:12.229036 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.229043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.229883 (+ 840us) server_negotiation.cc:407] Connection header received
0926 20:52:12.230078 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.230082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.230148 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.230235 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:12.232278 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.233629 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.235629 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.236187 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.236288 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.240189 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:12.240194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:12.240198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:12.240231 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:12.242580 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:12.243233 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:12.243237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:12.243239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:12.243360 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:12.243669 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:12.243672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:12.243674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:12.243940 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:12.244112 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.244459 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.244616 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:52:12.249177 10932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:52:12.256203 10932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:52:12.278698 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.268577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:36381 (local address 127.0.0.1:60520)
0926 20:52:12.270132 (+ 1555us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:12.270188 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:12.270486 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:12.271671 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:12.271680 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:12.271716 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:12.272119 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.272132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.274018 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.274024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.275927 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.275939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.277723 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.277730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.277943 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.278004 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 20:52:12.278112 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:52:12.278306 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1241,"thread_start_us":2839,"threads_started":1}
I20260926 20:52:12.279258 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.268788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.8.249.130:36381)
0926 20:52:12.269495 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.269507 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.270310 (+ 803us) server_negotiation.cc:407] Connection header received
0926 20:52:12.270815 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.270828 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.271180 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.271351 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:12.272333 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.273844 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.276089 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.277479 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.278341 (+ 862us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.278582 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.278653 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.278978 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":324,"threads_started":1}
I20260926 20:52:12.281213 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.269657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.8.249.131:46559)
0926 20:52:12.272840 (+ 3183us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.272849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.272880 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:52:12.272982 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.272994 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.273311 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.273492 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:12.274567 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.276297 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.278901 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.280106 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.280279 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.280534 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.280734 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.280896 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2966,"thread_start_us":2774,"threads_started":1}
I20260926 20:52:12.281229 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.268998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:46559 (local address 127.0.0.1:51548)
0926 20:52:12.271766 (+ 2768us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:12.271795 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:12.271949 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:12.273777 (+ 1828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:12.273781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:12.273810 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:12.274335 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.274349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.276443 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.276448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.278674 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.278694 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.280325 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.280331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.280508 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.280541 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 20:52:12.280640 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 20:52:12.280777 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2578,"mutex_wait_us":75,"thread_start_us":1296,"threads_started":1}
I20260926 20:52:12.281644 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.272430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:42499 (local address 127.0.0.1:53426)
0926 20:52:12.273002 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:12.273019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:12.273120 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:12.274285 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:12.274289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:12.274315 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:12.274708 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.274718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.276500 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.276507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.278500 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.278563 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.280686 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.280693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.280848 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.280878 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 20:52:12.280970 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:52:12.281112 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":394,"threads_started":1}
I20260926 20:52:12.281749 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.272105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53426 (local address 127.8.249.129:42499)
0926 20:52:12.273542 (+ 1437us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.273550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.273574 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:52:12.273660 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.273666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.273886 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.274070 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:12.274920 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.276371 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.278729 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.280205 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.280340 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.280543 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.281332 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.281488 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1292,"thread_start_us":1343,"threads_started":1}
I20260926 20:52:12.283396 11208 tablet_service.cc:1467] Processing CreateTablet for tablet ac92139b2f784cbb9582df2b0d6855af (DEFAULT_TABLE table=test-table [id=987ea1ca280c4ff6921033f1778a40a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:12.284883 11208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac92139b2f784cbb9582df2b0d6855af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:12.284758 11347 tablet_service.cc:1467] Processing CreateTablet for tablet ac92139b2f784cbb9582df2b0d6855af (DEFAULT_TABLE table=test-table [id=987ea1ca280c4ff6921033f1778a40a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:12.285104 11069 tablet_service.cc:1467] Processing CreateTablet for tablet ac92139b2f784cbb9582df2b0d6855af (DEFAULT_TABLE table=test-table [id=987ea1ca280c4ff6921033f1778a40a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:12.285969 11347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac92139b2f784cbb9582df2b0d6855af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:12.286306 11069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac92139b2f784cbb9582df2b0d6855af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:12.298242 11436 tablet_bootstrap.cc:492] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc: Bootstrap starting.
I20260926 20:52:12.299257 11437 tablet_bootstrap.cc:492] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32: Bootstrap starting.
I20260926 20:52:12.300758 11438 tablet_bootstrap.cc:492] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe: Bootstrap starting.
I20260926 20:52:12.302932 11437 tablet_bootstrap.cc:654] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:12.303748 11438 tablet_bootstrap.cc:654] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:12.303774 11436 tablet_bootstrap.cc:654] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:12.304262 11437 log.cc:824] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:12.304711 11438 log.cc:824] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:12.304987 11436 log.cc:824] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:12.307503 11437 tablet_bootstrap.cc:492] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32: No bootstrap required, opened a new log
I20260926 20:52:12.307631 11436 tablet_bootstrap.cc:492] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc: No bootstrap required, opened a new log
I20260926 20:52:12.307803 11437 ts_tablet_manager.cc:1397] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 20:52:12.307893 11436 ts_tablet_manager.cc:1397] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 20:52:12.309267 11438 tablet_bootstrap.cc:492] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe: No bootstrap required, opened a new log
I20260926 20:52:12.309522 11438 ts_tablet_manager.cc:1397] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260926 20:52:12.315078 11437 raft_consensus.cc:348] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } }
I20260926 20:52:12.315433 11437 raft_consensus.cc:374] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:12.315526 11437 raft_consensus.cc:729] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94682c641b8b457cbe5552c065e73a32, State: Initialized, Role: FOLLOWER
I20260926 20:52:12.316123 11437 consensus_queue.cc:260] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } }
I20260926 20:52:12.317137 11278 heartbeater.cc:503] Master 127.8.249.190:42267 was elected leader, sending a full tablet report...
I20260926 20:52:12.317801 11437 ts_tablet_manager.cc:1428] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260926 20:52:12.318491 11436 raft_consensus.cc:348] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } }
I20260926 20:52:12.318939 11436 raft_consensus.cc:374] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:12.319037 11436 raft_consensus.cc:729] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99df0a67d849469fa222ca16b8034ffc, State: Initialized, Role: FOLLOWER
I20260926 20:52:12.319602 11436 consensus_queue.cc:260] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } }
I20260926 20:52:12.319897 11438 raft_consensus.cc:348] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } }
I20260926 20:52:12.320344 11438 raft_consensus.cc:374] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:12.320462 11438 raft_consensus.cc:729] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbe00975fc74ad6b99f2b4140611cfe, State: Initialized, Role: FOLLOWER
I20260926 20:52:12.320843 11417 heartbeater.cc:503] Master 127.8.249.190:42267 was elected leader, sending a full tablet report...
I20260926 20:52:12.321010 11438 consensus_queue.cc:260] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } }
I20260926 20:52:12.321521 11436 ts_tablet_manager.cc:1428] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 20:52:12.324379 11438 ts_tablet_manager.cc:1428] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260926 20:52:12.371698 11140 tablet.cc:2369] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:52:12.401803 11279 tablet.cc:2369] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:52:12.428972 11418 tablet.cc:2369] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:52:12.458902 11442 raft_consensus.cc:482] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:12.459318 11442 raft_consensus.cc:504] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } }
I20260926 20:52:12.461643 11442 leader_election.cc:288] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99df0a67d849469fa222ca16b8034ffc (127.8.249.131:46559), 2bbe00975fc74ad6b99f2b4140611cfe (127.8.249.129:42499)
I20260926 20:52:12.470767 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.461960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:36169 (local address 127.8.249.131:46559)
0926 20:52:12.462189 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.462196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.463152 (+ 956us) server_negotiation.cc:407] Connection header received
0926 20:52:12.463394 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.463399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.463474 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.463564 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:12.464221 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.465268 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.468260 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.469475 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.469656 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.469732 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.470384 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.470524 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:52:12.471027 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.461986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:42499 (local address 127.8.249.130:33725)
0926 20:52:12.462648 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:12.462665 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:12.462855 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:12.463395 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:12.463402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:12.463428 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:12.463849 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.463864 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.465309 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.465313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.468070 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.468085 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.469758 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.469763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.470110 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.470247 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 20:52:12.470356 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 20:52:12.470511 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":312,"thread_start_us":352,"threads_started":1}
I20260926 20:52:12.470770 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.462500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:33725 (local address 127.8.249.129:42499)
0926 20:52:12.462774 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.462780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.462847 (+ 67us) server_negotiation.cc:407] Connection header received
0926 20:52:12.463017 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.463022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.463124 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.463213 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:12.464003 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.465159 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.468269 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.469486 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.469661 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.469732 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.470384 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.470516 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 20:52:12.471294 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.462088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:46559 (local address 127.8.249.130:36169)
0926 20:52:12.463060 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:12.463077 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:12.463250 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:12.463599 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:12.463607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:12.463626 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:12.464043 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.464055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.465397 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.465402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.468068 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.468085 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.469756 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.469763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.470108 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.470246 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 20:52:12.470345 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 20:52:12.470511 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":231,"thread_start_us":986,"threads_started":1}
I20260926 20:52:12.472092 11094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac92139b2f784cbb9582df2b0d6855af" candidate_uuid: "94682c641b8b457cbe5552c065e73a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" is_pre_election: true
I20260926 20:52:12.472414 11372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac92139b2f784cbb9582df2b0d6855af" candidate_uuid: "94682c641b8b457cbe5552c065e73a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99df0a67d849469fa222ca16b8034ffc" is_pre_election: true
I20260926 20:52:12.472582 11094 raft_consensus.cc:2454] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94682c641b8b457cbe5552c065e73a32 in term 0.
I20260926 20:52:12.472889 11372 raft_consensus.cc:2454] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94682c641b8b457cbe5552c065e73a32 in term 0.
I20260926 20:52:12.473299 11159 leader_election.cc:302] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bbe00975fc74ad6b99f2b4140611cfe, 94682c641b8b457cbe5552c065e73a32; no voters:
I20260926 20:52:12.473711 11442 raft_consensus.cc:2781] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:52:12.473855 11442 raft_consensus.cc:482] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:12.473975 11442 raft_consensus.cc:3037] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:12.476176 11442 raft_consensus.cc:504] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } }
I20260926 20:52:12.476861 11442 leader_election.cc:288] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [CANDIDATE]: Term 1 election: Requested vote from peers 99df0a67d849469fa222ca16b8034ffc (127.8.249.131:46559), 2bbe00975fc74ad6b99f2b4140611cfe (127.8.249.129:42499)
I20260926 20:52:12.477288 11372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac92139b2f784cbb9582df2b0d6855af" candidate_uuid: "94682c641b8b457cbe5552c065e73a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99df0a67d849469fa222ca16b8034ffc"
I20260926 20:52:12.477397 11094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac92139b2f784cbb9582df2b0d6855af" candidate_uuid: "94682c641b8b457cbe5552c065e73a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bbe00975fc74ad6b99f2b4140611cfe"
I20260926 20:52:12.477510 11372 raft_consensus.cc:3037] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:12.477607 11094 raft_consensus.cc:3037] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:12.480759 11372 raft_consensus.cc:2454] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94682c641b8b457cbe5552c065e73a32 in term 1.
I20260926 20:52:12.480759 11094 raft_consensus.cc:2454] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94682c641b8b457cbe5552c065e73a32 in term 1.
I20260926 20:52:12.481154 11159 leader_election.cc:302] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bbe00975fc74ad6b99f2b4140611cfe, 94682c641b8b457cbe5552c065e73a32; no voters:
I20260926 20:52:12.481455 11442 raft_consensus.cc:2781] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:12.481894 11442 raft_consensus.cc:686] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [term 1 LEADER]: Becoming Leader. State: Replica: 94682c641b8b457cbe5552c065e73a32, State: Running, Role: LEADER
I20260926 20:52:12.482365 11442 consensus_queue.cc:237] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } }
I20260926 20:52:12.486649 10932 catalog_manager.cc:5495] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94682c641b8b457cbe5552c065e73a32 (127.8.249.130). New cstate: current_term: 1 leader_uuid: "94682c641b8b457cbe5552c065e73a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:52:12 dist-test-slave-2dg3 krb5kdc[10887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455930, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:12.551118 11454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.537310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42267 (local address 127.0.0.1:49604)
0926 20:52:12.537963 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:12.537984 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:12.538112 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:12.538426 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:12.538429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:12.539178 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:12.539431 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.539441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.540610 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.540613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.541545 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.541556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.542337 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.542341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.542473 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.543155 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:12.543172 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:12.545382 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:12.548489 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:12.548498 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:12.548505 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:12.548910 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:12.549248 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:12.549251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:12.549253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:12.549361 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:12.549854 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:12.549860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:12.550016 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 20:52:12.550235 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 20:52:12.550574 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":358,"threads_started":1}
I20260926 20:52:12.551236 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.537285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.8.249.190:42267)
0926 20:52:12.537472 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.537477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.538066 (+ 589us) server_negotiation.cc:407] Connection header received
0926 20:52:12.538210 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.538213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.538292 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.538386 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:12.539616 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.540500 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.541684 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.542116 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.542284 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.545566 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:12.545572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:12.545575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:12.545606 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:12.548367 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:12.549010 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:12.549013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:12.549016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:12.549142 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:12.549455 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:12.549458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:12.549460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:12.549744 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:12.549867 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.550378 (+ 511us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.550767 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":71}
I20260926 20:52:12.556161 10932 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:49604:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:52:12.557099 10932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:52:12.563397 11208 tablet_service.cc:1467] Processing CreateTablet for tablet d613c57cba0646b2ba43cfb79df63bff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2840343f9e94484d83b92735466370bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:52:12.563361 11069 tablet_service.cc:1467] Processing CreateTablet for tablet d613c57cba0646b2ba43cfb79df63bff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2840343f9e94484d83b92735466370bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:52:12.563598 11347 tablet_service.cc:1467] Processing CreateTablet for tablet d613c57cba0646b2ba43cfb79df63bff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2840343f9e94484d83b92735466370bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:52:12.564342 11208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d613c57cba0646b2ba43cfb79df63bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:12.564353 11069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d613c57cba0646b2ba43cfb79df63bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:12.564369 11347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d613c57cba0646b2ba43cfb79df63bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:12.570119 11437 tablet_bootstrap.cc:492] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32: Bootstrap starting.
I20260926 20:52:12.571424 11438 tablet_bootstrap.cc:492] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe: Bootstrap starting.
I20260926 20:52:12.571547 11437 tablet_bootstrap.cc:654] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:12.572943 11438 tablet_bootstrap.cc:654] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:12.573925 11437 tablet_bootstrap.cc:492] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32: No bootstrap required, opened a new log
I20260926 20:52:12.573971 11436 tablet_bootstrap.cc:492] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc: Bootstrap starting.
I20260926 20:52:12.574072 11437 ts_tablet_manager.cc:1397] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 20:52:12.574873 11437 raft_consensus.cc:348] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.575145 11437 raft_consensus.cc:374] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:12.575237 11437 raft_consensus.cc:729] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94682c641b8b457cbe5552c065e73a32, State: Initialized, Role: FOLLOWER
I20260926 20:52:12.575596 11436 tablet_bootstrap.cc:654] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:12.575500 11437 consensus_queue.cc:260] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.576177 11437 ts_tablet_manager.cc:1428] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 20:52:12.576241 11438 tablet_bootstrap.cc:492] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe: No bootstrap required, opened a new log
I20260926 20:52:12.576038 11442 tablet_replica.cc:439] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } }
I20260926 20:52:12.576388 11442 tablet_replica.cc:442] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:12.576412 11438 ts_tablet_manager.cc:1397] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 20:52:12.577232 11438 raft_consensus.cc:348] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.577543 11438 raft_consensus.cc:374] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:12.577661 11438 raft_consensus.cc:729] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbe00975fc74ad6b99f2b4140611cfe, State: Initialized, Role: FOLLOWER
I20260926 20:52:12.577930 11438 consensus_queue.cc:260] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.578609 11438 ts_tablet_manager.cc:1428] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 20:52:12.578428 11444 tablet_replica.cc:439] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } }
I20260926 20:52:12.578872 11444 tablet_replica.cc:442] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:12.579294 11436 tablet_bootstrap.cc:492] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc: No bootstrap required, opened a new log
I20260926 20:52:12.579438 11436 ts_tablet_manager.cc:1397] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 20:52:12.580253 11436 raft_consensus.cc:348] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.580513 11436 raft_consensus.cc:374] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:12.580612 11436 raft_consensus.cc:729] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99df0a67d849469fa222ca16b8034ffc, State: Initialized, Role: FOLLOWER
I20260926 20:52:12.580832 11436 consensus_queue.cc:260] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.581401 11436 ts_tablet_manager.cc:1428] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 20:52:12.581282 11443 tablet_replica.cc:439] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } }
I20260926 20:52:12.581684 11443 tablet_replica.cc:442] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:12.708178 11444 raft_consensus.cc:482] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:12.708389 11444 raft_consensus.cc:504] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.709992 11444 leader_election.cc:288] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94682c641b8b457cbe5552c065e73a32 (127.8.249.130:36381), 99df0a67d849469fa222ca16b8034ffc (127.8.249.131:46559)
I20260926 20:52:12.717132 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.710432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:58007 (local address 127.8.249.131:46559)
0926 20:52:12.710627 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.710633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.711562 (+ 929us) server_negotiation.cc:407] Connection header received
0926 20:52:12.711768 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.711772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.711862 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.711951 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:12.712605 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.713305 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.715388 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.716113 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.716290 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.716363 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.716722 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.716850 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:52:12.717141 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.710570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:57333 (local address 127.8.249.130:36381)
0926 20:52:12.710847 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.710856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.711446 (+ 590us) server_negotiation.cc:407] Connection header received
0926 20:52:12.711670 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.711676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.711762 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.711855 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:12.712502 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.713277 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.715284 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.716096 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.716278 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.716365 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.716726 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.716869 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 20:52:12.717305 11458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.710394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:36381 (local address 127.8.249.129:57333)
0926 20:52:12.711318 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:12.711334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:12.711473 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:12.711886 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:12.711894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:12.711932 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:12.712335 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.712344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.713418 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.713422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.715094 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.715106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.716328 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.716334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.716508 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.716590 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 20:52:12.716695 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 20:52:12.716849 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":88,"thread_start_us":514,"threads_started":1}
I20260926 20:52:12.717439 11459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.710585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:46559 (local address 127.8.249.129:58007)
0926 20:52:12.711431 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:12.711456 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:12.711626 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:12.711998 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:12.712002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:12.712043 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:12.712409 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.712440 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.713470 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.713473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.715246 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.715261 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.716364 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.716370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.716552 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.716593 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 20:52:12.716697 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:52:12.716857 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":50,"thread_start_us":504,"threads_started":1}
I20260926 20:52:12.718078 11372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d613c57cba0646b2ba43cfb79df63bff" candidate_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99df0a67d849469fa222ca16b8034ffc" is_pre_election: true
I20260926 20:52:12.718358 11372 raft_consensus.cc:2454] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bbe00975fc74ad6b99f2b4140611cfe in term 0.
I20260926 20:52:12.718400 11232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d613c57cba0646b2ba43cfb79df63bff" candidate_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94682c641b8b457cbe5552c065e73a32" is_pre_election: true
I20260926 20:52:12.718886 11232 raft_consensus.cc:2454] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bbe00975fc74ad6b99f2b4140611cfe in term 0.
I20260926 20:52:12.718916 11022 leader_election.cc:302] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bbe00975fc74ad6b99f2b4140611cfe, 99df0a67d849469fa222ca16b8034ffc; no voters:
I20260926 20:52:12.719377 11444 raft_consensus.cc:2781] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:52:12.719524 11444 raft_consensus.cc:482] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:12.719632 11444 raft_consensus.cc:3037] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:12.721850 11444 raft_consensus.cc:504] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.722460 11444 leader_election.cc:288] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [CANDIDATE]: Term 1 election: Requested vote from peers 94682c641b8b457cbe5552c065e73a32 (127.8.249.130:36381), 99df0a67d849469fa222ca16b8034ffc (127.8.249.131:46559)
I20260926 20:52:12.722942 11232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d613c57cba0646b2ba43cfb79df63bff" candidate_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94682c641b8b457cbe5552c065e73a32"
I20260926 20:52:12.722955 11372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d613c57cba0646b2ba43cfb79df63bff" candidate_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99df0a67d849469fa222ca16b8034ffc"
I20260926 20:52:12.723158 11232 raft_consensus.cc:3037] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:12.723191 11372 raft_consensus.cc:3037] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:12.725373 11372 raft_consensus.cc:2454] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bbe00975fc74ad6b99f2b4140611cfe in term 1.
I20260926 20:52:12.725790 11232 raft_consensus.cc:2454] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bbe00975fc74ad6b99f2b4140611cfe in term 1.
I20260926 20:52:12.725818 11022 leader_election.cc:302] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bbe00975fc74ad6b99f2b4140611cfe, 99df0a67d849469fa222ca16b8034ffc; no voters:
I20260926 20:52:12.726119 11444 raft_consensus.cc:2781] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:12.726573 11444 raft_consensus.cc:686] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 LEADER]: Becoming Leader. State: Replica: 2bbe00975fc74ad6b99f2b4140611cfe, State: Running, Role: LEADER
I20260926 20:52:12.727169 11444 consensus_queue.cc:237] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.730526 11460 tablet_replica.cc:439] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe: TxnStatusTablet state changed. Reason: New leader 2bbe00975fc74ad6b99f2b4140611cfe. Latest consensus state: current_term: 1 leader_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } }
I20260926 20:52:12.730986 11460 tablet_replica.cc:442] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:52:12.731681 11462 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:52:12.731954 11462 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:52:12.731988 10930 catalog_manager.cc:5495] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bbe00975fc74ad6b99f2b4140611cfe (127.8.249.129). New cstate: current_term: 1 leader_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } health_report { overall_health: UNKNOWN } } }
I20260926 20:52:12.771999 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.765784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.8.249.129:42499)
0926 20:52:12.765983 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.765990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.766119 (+ 129us) server_negotiation.cc:407] Connection header received
0926 20:52:12.766400 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.766403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.766483 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.766607 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:12.767031 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.767853 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.769839 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.770256 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.770352 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.770917 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:12.771447 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:12.771563 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.771696 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.771760 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 20:52:12.772140 11454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.765763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:42499 (local address 127.0.0.1:53430)
0926 20:52:12.766028 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:12.766048 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:12.766198 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:12.766654 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:12.766662 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:12.766676 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:52:12.766909 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.766922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.767982 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.767990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.769716 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.769729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.770399 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.770406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.770615 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.770686 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:52:12.771561 (+ 875us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:52:12.771582 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:52:12.771672 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:52:12.771828 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 20:52:12.779698 11232 raft_consensus.cc:1261] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Refusing update from remote peer 2bbe00975fc74ad6b99f2b4140611cfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:12.779723 11372 raft_consensus.cc:1261] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Refusing update from remote peer 2bbe00975fc74ad6b99f2b4140611cfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:12.780656 11460 consensus_queue.cc:1035] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:12.781013 11444 consensus_queue.cc:1035] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:12.788730 11442 tablet_replica.cc:439] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32: TxnStatusTablet state changed. Reason: New leader 2bbe00975fc74ad6b99f2b4140611cfe. Latest consensus state: current_term: 1 leader_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } }
I20260926 20:52:12.789069 11442 tablet_replica.cc:442] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:12.793473 11443 tablet_replica.cc:439] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc: TxnStatusTablet state changed. Reason: New leader 2bbe00975fc74ad6b99f2b4140611cfe. Latest consensus state: current_term: 1 leader_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } }
I20260926 20:52:12.793732 11443 tablet_replica.cc:442] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:12.793704 11465 mvcc.cc:205] Tried to move back new op lower bound from 7333707500655243264 to 7333707500454354944. Current Snapshot: MvccSnapshot[applied={T|T < 7333707500655243264}]
I20260926 20:52:12.799316 11460 tablet_replica.cc:439] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } }
I20260926 20:52:12.799700 11460 tablet_replica.cc:442] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:52:12.803049 11460 tablet_replica.cc:439] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } }
I20260926 20:52:12.803412 11460 tablet_replica.cc:442] T d613c57cba0646b2ba43cfb79df63bff P 2bbe00975fc74ad6b99f2b4140611cfe: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:52:12.807709 11443 tablet_replica.cc:439] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } }
I20260926 20:52:12.807962 11443 tablet_replica.cc:442] T d613c57cba0646b2ba43cfb79df63bff P 99df0a67d849469fa222ca16b8034ffc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:12.807914 11442 tablet_replica.cc:439] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } } }
I20260926 20:52:12.808178 11442 tablet_replica.cc:442] T d613c57cba0646b2ba43cfb79df63bff P 94682c641b8b457cbe5552c065e73a32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:52:12.820360 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.812381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.8.249.130:36381)
0926 20:52:12.812566 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.812572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.814084 (+ 1512us) server_negotiation.cc:407] Connection header received
0926 20:52:12.814283 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.814288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.814413 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.814593 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:12.815053 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.816164 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.817603 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.818247 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.818374 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.818975 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:12.819740 (+ 765us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:12.819882 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.819936 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.820028 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:52:12.837652 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.831488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:37787 (local address 127.8.249.129:42499)
0926 20:52:12.831700 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.831707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.832219 (+ 512us) server_negotiation.cc:407] Connection header received
0926 20:52:12.832437 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.832441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.832561 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.832651 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:12.833177 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.834064 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.835829 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.836251 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.836419 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.836868 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:12.836970 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:12.837075 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.837227 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.837304 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:52:12.837709 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.831458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:42499 (local address 127.8.249.130:37787)
0926 20:52:12.832113 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:12.832139 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:12.832260 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:12.832680 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:12.832684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:12.832698 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:52:12.833013 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.833024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.834189 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.834195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.835677 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.835690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.836457 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.836465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.836595 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.836652 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:52:12.837075 (+ 423us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:52:12.837097 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:52:12.837193 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 20:52:12.837329 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":340,"threads_started":1}
I20260926 20:52:12.848043 11372 raft_consensus.cc:1261] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Refusing update from remote peer 94682c641b8b457cbe5552c065e73a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:12.848165 11094 raft_consensus.cc:1261] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Refusing update from remote peer 94682c641b8b457cbe5552c065e73a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:12.848666 11447 consensus_queue.cc:1035] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:12.849033 11442 consensus_queue.cc:1035] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:12.860432 11466 mvcc.cc:205] Tried to move back new op lower bound from 7333707500938444800 to 7333707499451568128. Current Snapshot: MvccSnapshot[applied={T|T < 7333707500938444800}]
I20260926 20:52:12.888108 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.881646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:37157 (local address 127.8.249.130:36381)
0926 20:52:12.881847 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:12.881853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:12.882513 (+ 660us) server_negotiation.cc:407] Connection header received
0926 20:52:12.882739 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:12.882744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:12.882876 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:12.882975 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:12.883627 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.884709 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.886134 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.886562 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.886703 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.887260 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:12.887431 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:12.887525 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:12.887695 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 20:52:12.887776 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:52:12.888301 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:12.881683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:36381 (local address 127.8.249.129:37157)
0926 20:52:12.882426 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:12.882449 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:12.882594 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:12.883008 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:12.883013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:12.883026 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:52:12.883419 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.883432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.884825 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.884833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.885981 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:12.885994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:12.886741 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:12.886748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:12.886985 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:12.887060 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:52:12.887537 (+ 477us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:52:12.887564 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:52:12.887656 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:52:12.887812 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":344,"threads_started":1}
I20260926 20:52:12.933380 10931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:12.933669 10931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:12.937939 10931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4: Sending DeleteTablet for 3 replicas of tablet ac92139b2f784cbb9582df2b0d6855af
I20260926 20:52:12.939062 11208 tablet_service.cc:1514] Processing DeleteTablet for tablet ac92139b2f784cbb9582df2b0d6855af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:12 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:60520
I20260926 20:52:12.939827 11069 tablet_service.cc:1514] Processing DeleteTablet for tablet ac92139b2f784cbb9582df2b0d6855af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:12 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:53426
I20260926 20:52:12.940151 11347 tablet_service.cc:1514] Processing DeleteTablet for tablet ac92139b2f784cbb9582df2b0d6855af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:12 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:51548
I20260926 20:52:12.943190 10930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:52:12.943872 10930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:52:12.952431 11068 tablet_service.cc:1467] Processing CreateTablet for tablet c3b737d96a02485eaea84e42b7c1c3b7 (DEFAULT_TABLE table=test-table [id=67fcd95e6e054995b99ba52d926ff119]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:12.952817 11346 tablet_service.cc:1467] Processing CreateTablet for tablet c3b737d96a02485eaea84e42b7c1c3b7 (DEFAULT_TABLE table=test-table [id=67fcd95e6e054995b99ba52d926ff119]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:12.953256 11068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b737d96a02485eaea84e42b7c1c3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:12.953413 11346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b737d96a02485eaea84e42b7c1c3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:12.960814 11436 tablet_bootstrap.cc:492] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc: Bootstrap starting.
I20260926 20:52:12.961944 11489 tablet_replica.cc:330] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc: stopping tablet replica
I20260926 20:52:12.962544 11489 raft_consensus.cc:2229] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:12.963155 11489 raft_consensus.cc:2258] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:12.963356 11436 tablet_bootstrap.cc:654] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:12.963979 11488 tablet_replica.cc:330] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe: stopping tablet replica
I20260926 20:52:12.964447 11488 raft_consensus.cc:2229] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:12.964520 11207 tablet_service.cc:1467] Processing CreateTablet for tablet c3b737d96a02485eaea84e42b7c1c3b7 (DEFAULT_TABLE table=test-table [id=67fcd95e6e054995b99ba52d926ff119]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:12.964805 11488 raft_consensus.cc:2258] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:12.965152 11207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b737d96a02485eaea84e42b7c1c3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:12.965554 11487 tablet_replica.cc:330] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32: stopping tablet replica
I20260926 20:52:12.965742 11489 ts_tablet_manager.cc:1905] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:12.965957 11487 raft_consensus.cc:2229] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:52:12.966547 11487 raft_consensus.cc:2258] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:12.968212 11488 ts_tablet_manager.cc:1905] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:12.972740 11489 ts_tablet_manager.cc:1918] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:52:12.972908 11489 log.cc:1196] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/ac92139b2f784cbb9582df2b0d6855af
I20260926 20:52:12.973094 11487 ts_tablet_manager.cc:1905] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:12.973425 11489 ts_tablet_manager.cc:1939] T ac92139b2f784cbb9582df2b0d6855af P 99df0a67d849469fa222ca16b8034ffc: Deleting consensus metadata
I20260926 20:52:12.975271 10919 catalog_manager.cc:4841] TS 99df0a67d849469fa222ca16b8034ffc (127.8.249.131:46559): tablet ac92139b2f784cbb9582df2b0d6855af (table test-table [id=987ea1ca280c4ff6921033f1778a40a0]) successfully deleted
I20260926 20:52:12.975714 11437 tablet_bootstrap.cc:492] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32: Bootstrap starting.
I20260926 20:52:12.976089 11438 tablet_bootstrap.cc:492] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe: Bootstrap starting.
I20260926 20:52:12.976855 11436 tablet_bootstrap.cc:492] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc: No bootstrap required, opened a new log
I20260926 20:52:12.977017 11436 ts_tablet_manager.cc:1397] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20260926 20:52:12.977411 11437 tablet_bootstrap.cc:654] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:12.977802 11438 tablet_bootstrap.cc:654] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:12.977815 11436 raft_consensus.cc:348] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.978048 11436 raft_consensus.cc:374] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:12.978127 11436 raft_consensus.cc:729] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99df0a67d849469fa222ca16b8034ffc, State: Initialized, Role: FOLLOWER
I20260926 20:52:12.978433 11436 consensus_queue.cc:260] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.981765 11436 ts_tablet_manager.cc:1428] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 20:52:12.988355 11438 tablet_bootstrap.cc:492] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe: No bootstrap required, opened a new log
I20260926 20:52:12.988610 11438 ts_tablet_manager.cc:1397] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.006s
I20260926 20:52:12.989745 11438 raft_consensus.cc:348] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.990077 11438 raft_consensus.cc:374] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:12.990203 11438 raft_consensus.cc:729] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbe00975fc74ad6b99f2b4140611cfe, State: Initialized, Role: FOLLOWER
I20260926 20:52:12.990554 11438 consensus_queue.cc:260] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.993922 11438 ts_tablet_manager.cc:1428] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260926 20:52:12.997263 11437 tablet_bootstrap.cc:492] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32: No bootstrap required, opened a new log
I20260926 20:52:12.997565 11437 ts_tablet_manager.cc:1397] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.011s
I20260926 20:52:12.998288 11487 ts_tablet_manager.cc:1918] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:52:12.998473 11487 log.cc:1196] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/ac92139b2f784cbb9582df2b0d6855af
I20260926 20:52:12.998590 11437 raft_consensus.cc:348] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:12.998945 11437 raft_consensus.cc:374] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:12.999886 11488 ts_tablet_manager.cc:1918] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:52:13.000061 11488 log.cc:1196] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/ac92139b2f784cbb9582df2b0d6855af
I20260926 20:52:13.000209 11487 ts_tablet_manager.cc:1939] T ac92139b2f784cbb9582df2b0d6855af P 94682c641b8b457cbe5552c065e73a32: Deleting consensus metadata
I20260926 20:52:13.000656 11488 ts_tablet_manager.cc:1939] T ac92139b2f784cbb9582df2b0d6855af P 2bbe00975fc74ad6b99f2b4140611cfe: Deleting consensus metadata
I20260926 20:52:13.001925 11437 raft_consensus.cc:729] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94682c641b8b457cbe5552c065e73a32, State: Initialized, Role: FOLLOWER
I20260926 20:52:13.002007 10916 catalog_manager.cc:4841] TS 94682c641b8b457cbe5552c065e73a32 (127.8.249.130:36381): tablet ac92139b2f784cbb9582df2b0d6855af (table test-table [id=987ea1ca280c4ff6921033f1778a40a0]) successfully deleted
I20260926 20:52:13.002260 11437 consensus_queue.cc:260] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:13.002671 10916 catalog_manager.cc:4841] TS 2bbe00975fc74ad6b99f2b4140611cfe (127.8.249.129:42499): tablet ac92139b2f784cbb9582df2b0d6855af (table test-table [id=987ea1ca280c4ff6921033f1778a40a0]) successfully deleted
I20260926 20:52:13.003626 11437 ts_tablet_manager.cc:1428] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260926 20:52:13.117564 11443 raft_consensus.cc:482] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:13.117798 11443 raft_consensus.cc:504] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:13.119695 11443 leader_election.cc:288] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bbe00975fc74ad6b99f2b4140611cfe (127.8.249.129:42499), 94682c641b8b457cbe5552c065e73a32 (127.8.249.130:36381)
I20260926 20:52:13.147253 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:13.132137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:50575 (local address 127.8.249.130:36381)
0926 20:52:13.132343 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:13.132361 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:13.139304 (+ 6943us) server_negotiation.cc:407] Connection header received
0926 20:52:13.139802 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:13.139809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:13.139895 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:13.139964 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:13.140824 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.141943 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.144507 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.145902 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.146057 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:13.146140 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:13.146771 (+ 631us) server_negotiation.cc:299] Negotiation successful
0926 20:52:13.146926 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:52:13.148397 11494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:13.132508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:36381 (local address 127.8.249.131:50575)
0926 20:52:13.139193 (+ 6685us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:13.139215 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:13.139572 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:13.140165 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:13.140170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:13.140195 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:13.140578 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:13.140591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.142156 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.142161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:13.144303 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:13.144319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.146372 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.146378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:13.146561 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:13.146600 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 20:52:13.147816 (+ 1216us) client_negotiation.cc:240] Negotiation successful
0926 20:52:13.147960 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6545,"mutex_wait_us":3,"thread_start_us":6811,"threads_started":1}
I20260926 20:52:13.148690 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:13.123103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:41637 (local address 127.8.249.129:42499)
0926 20:52:13.123288 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:13.123294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:13.131330 (+ 8036us) server_negotiation.cc:407] Connection header received
0926 20:52:13.131607 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:13.131616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:13.131693 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:13.131785 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:13.139141 (+ 7356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.140367 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.145931 (+ 5564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.147383 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.147570 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:13.147644 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:13.148232 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 20:52:13.148369 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:52:13.149451 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:13.120245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:42499 (local address 127.8.249.131:41637)
0926 20:52:13.131203 (+ 10958us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:13.131228 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:13.131434 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:13.136101 (+ 4667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:13.136107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:13.136139 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:13.136544 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:13.136569 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.140620 (+ 4051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.140628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:13.145757 (+ 5129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:13.145771 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.147854 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.147861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:13.148042 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:13.148075 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 20:52:13.148959 (+ 884us) client_negotiation.cc:240] Negotiation successful
0926 20:52:13.149099 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10803,"mutex_wait_us":2970,"thread_start_us":11305,"threads_started":1}
I20260926 20:52:13.149694 11232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b737d96a02485eaea84e42b7c1c3b7" candidate_uuid: "99df0a67d849469fa222ca16b8034ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94682c641b8b457cbe5552c065e73a32" is_pre_election: true
I20260926 20:52:13.149991 11232 raft_consensus.cc:2454] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99df0a67d849469fa222ca16b8034ffc in term 0.
I20260926 20:52:13.150157 11094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b737d96a02485eaea84e42b7c1c3b7" candidate_uuid: "99df0a67d849469fa222ca16b8034ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" is_pre_election: true
I20260926 20:52:13.150399 11094 raft_consensus.cc:2454] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99df0a67d849469fa222ca16b8034ffc in term 0.
I20260926 20:52:13.150554 11298 leader_election.cc:302] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94682c641b8b457cbe5552c065e73a32, 99df0a67d849469fa222ca16b8034ffc; no voters:
I20260926 20:52:13.151403 11443 raft_consensus.cc:2781] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:52:13.151535 11443 raft_consensus.cc:482] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:13.151635 11443 raft_consensus.cc:3037] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:13.154776 11443 raft_consensus.cc:504] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:13.155462 11443 leader_election.cc:288] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [CANDIDATE]: Term 1 election: Requested vote from peers 2bbe00975fc74ad6b99f2b4140611cfe (127.8.249.129:42499), 94682c641b8b457cbe5552c065e73a32 (127.8.249.130:36381)
I20260926 20:52:13.156255 11094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b737d96a02485eaea84e42b7c1c3b7" candidate_uuid: "99df0a67d849469fa222ca16b8034ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bbe00975fc74ad6b99f2b4140611cfe"
I20260926 20:52:13.156476 11094 raft_consensus.cc:3037] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:13.159250 11232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b737d96a02485eaea84e42b7c1c3b7" candidate_uuid: "99df0a67d849469fa222ca16b8034ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94682c641b8b457cbe5552c065e73a32"
I20260926 20:52:13.159502 11232 raft_consensus.cc:3037] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:13.162775 11232 raft_consensus.cc:2454] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99df0a67d849469fa222ca16b8034ffc in term 1.
I20260926 20:52:13.164017 11094 raft_consensus.cc:2454] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99df0a67d849469fa222ca16b8034ffc in term 1.
I20260926 20:52:13.164510 11298 leader_election.cc:302] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bbe00975fc74ad6b99f2b4140611cfe, 99df0a67d849469fa222ca16b8034ffc; no voters:
I20260926 20:52:13.165092 11443 raft_consensus.cc:2781] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:13.168084 11443 raft_consensus.cc:686] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [term 1 LEADER]: Becoming Leader. State: Replica: 99df0a67d849469fa222ca16b8034ffc, State: Running, Role: LEADER
I20260926 20:52:13.168531 11443 consensus_queue.cc:237] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:13.172523 10930 catalog_manager.cc:5495] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc reported cstate change: term changed from 0 to 1, leader changed from <none> to 99df0a67d849469fa222ca16b8034ffc (127.8.249.131). New cstate: current_term: 1 leader_uuid: "99df0a67d849469fa222ca16b8034ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } health_report { overall_health: HEALTHY } } }
I20260926 20:52:13.235144 11497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:13.227104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.8.249.131:46559)
0926 20:52:13.227585 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:13.227591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:13.227609 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:52:13.227707 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:13.227714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:13.227972 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:13.228598 (+ 626us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:13.228740 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.229833 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.231735 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.232474 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.232614 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:13.233100 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:13.233911 (+ 811us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:13.234582 (+ 671us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:13.234634 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 20:52:13.234748 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":210,"threads_started":1}
I20260926 20:52:13.252696 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:13.246716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:42499 (local address 127.8.249.131:45217)
0926 20:52:13.247480 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:13.247496 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:13.247631 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:13.248022 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:13.248029 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:13.248049 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:52:13.248297 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:13.248304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.249661 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.249665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:13.250709 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:13.250720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.251663 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.251667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:13.251792 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:13.251857 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:52:13.252252 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:52:13.252260 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:52:13.252319 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 20:52:13.252416 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":410,"threads_started":1}
I20260926 20:52:13.252733 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:13.246748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:45217 (local address 127.8.249.129:42499)
0926 20:52:13.246956 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:13.246962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:13.247593 (+ 631us) server_negotiation.cc:407] Connection header received
0926 20:52:13.247769 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:13.247775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:13.247860 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:13.247986 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:13.248447 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.249531 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.250879 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.251488 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.251631 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:13.252019 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:13.252156 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:13.252252 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:13.252370 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 20:52:13.252448 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 20:52:13.261777 11094 raft_consensus.cc:1261] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Refusing update from remote peer 99df0a67d849469fa222ca16b8034ffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:13.261773 11231 raft_consensus.cc:1261] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Refusing update from remote peer 99df0a67d849469fa222ca16b8034ffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:13.262503 11495 consensus_queue.cc:1035] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:13.262959 11443 consensus_queue.cc:1035] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:13.268707 11467 mvcc.cc:205] Tried to move back new op lower bound from 7333707502634119168 to 7333707502262976512. Current Snapshot: MvccSnapshot[applied={T|T < 7333707502634119168}]
I20260926 20:52:13.294991 11497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:13.289365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:40001 (local address 127.8.249.131:46559)
0926 20:52:13.289532 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:13.289537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:13.289554 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:52:13.289612 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:13.289616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:13.289687 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:13.289793 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:13.290186 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.291368 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.293305 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.293718 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.293812 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:13.294256 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:13.294355 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:13.294421 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:13.294624 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 20:52:13.294691 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:52:13.295032 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:13.288563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:46559 (local address 127.8.249.129:40001)
0926 20:52:13.288901 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:13.288922 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:13.289052 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:13.289812 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:13.289817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:13.289828 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:52:13.290025 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:13.290036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.291553 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.291559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:13.293157 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:13.293172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:13.293877 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:13.293883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:13.294063 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:13.294114 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:52:13.294459 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:52:13.294470 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 20:52:13.294568 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:52:13.294713 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
W20260926 20:52:13.296160 11467 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:52:14.912668 10930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:14.912883 10930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:14.916976 10930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4: Sending DeleteTablet for 3 replicas of tablet c3b737d96a02485eaea84e42b7c1c3b7
I20260926 20:52:14.917775 11068 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b737d96a02485eaea84e42b7c1c3b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:14 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:53426
I20260926 20:52:14.917845 11347 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b737d96a02485eaea84e42b7c1c3b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:14 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:51548
I20260926 20:52:14.917889 11207 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b737d96a02485eaea84e42b7c1c3b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:14 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:60520
I20260926 20:52:14.929315 11530 tablet_replica.cc:330] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32: stopping tablet replica
I20260926 20:52:14.929836 11530 raft_consensus.cc:2229] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:14.930264 11530 raft_consensus.cc:2258] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:14.931641 11530 ts_tablet_manager.cc:1905] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:14.935309 11529 tablet_replica.cc:330] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc: stopping tablet replica
I20260926 20:52:14.935556 11528 tablet_replica.cc:330] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe: stopping tablet replica
I20260926 20:52:14.935572 11529 raft_consensus.cc:2229] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:52:14.935763 11528 raft_consensus.cc:2229] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:14.936024 11528 raft_consensus.cc:2258] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:14.936329 11529 raft_consensus.cc:2258] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:14.937026 11528 ts_tablet_manager.cc:1905] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:14.937378 11529 ts_tablet_manager.cc:1905] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:14.941210 11530 ts_tablet_manager.cc:1918] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:52:14.941424 11530 log.cc:1196] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/c3b737d96a02485eaea84e42b7c1c3b7
I20260926 20:52:14.941941 11530 ts_tablet_manager.cc:1939] T c3b737d96a02485eaea84e42b7c1c3b7 P 94682c641b8b457cbe5552c065e73a32: Deleting consensus metadata
I20260926 20:52:14.943236 10916 catalog_manager.cc:4841] TS 94682c641b8b457cbe5552c065e73a32 (127.8.249.130:36381): tablet c3b737d96a02485eaea84e42b7c1c3b7 (table test-table [id=67fcd95e6e054995b99ba52d926ff119]) successfully deleted
I20260926 20:52:14.943955 11528 ts_tablet_manager.cc:1918] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:52:14.944121 11528 log.cc:1196] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/c3b737d96a02485eaea84e42b7c1c3b7
I20260926 20:52:14.944562 11528 ts_tablet_manager.cc:1939] T c3b737d96a02485eaea84e42b7c1c3b7 P 2bbe00975fc74ad6b99f2b4140611cfe: Deleting consensus metadata
I20260926 20:52:14.945164 11529 ts_tablet_manager.cc:1918] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:52:14.945331 11529 log.cc:1196] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/c3b737d96a02485eaea84e42b7c1c3b7
I20260926 20:52:14.945595 10916 catalog_manager.cc:4841] TS 2bbe00975fc74ad6b99f2b4140611cfe (127.8.249.129:42499): tablet c3b737d96a02485eaea84e42b7c1c3b7 (table test-table [id=67fcd95e6e054995b99ba52d926ff119]) successfully deleted
I20260926 20:52:14.945865 11529 ts_tablet_manager.cc:1939] T c3b737d96a02485eaea84e42b7c1c3b7 P 99df0a67d849469fa222ca16b8034ffc: Deleting consensus metadata
I20260926 20:52:14.947094 10919 catalog_manager.cc:4841] TS 99df0a67d849469fa222ca16b8034ffc (127.8.249.131:46559): tablet c3b737d96a02485eaea84e42b7c1c3b7 (table test-table [id=67fcd95e6e054995b99ba52d926ff119]) successfully deleted
Sep 26 20:52:14 dist-test-slave-2dg3 krb5kdc[10887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:52:14 dist-test-slave-2dg3 krb5kdc[10887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:14.978883 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:14.964796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.8.249.190:42267)
0926 20:52:14.965379 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:14.965387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:14.965488 (+ 101us) server_negotiation.cc:407] Connection header received
0926 20:52:14.965715 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:14.965721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:14.965796 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:14.965912 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:14.967357 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:14.968147 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:14.969375 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:14.969774 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:14.969896 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:14.973821 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:14.973826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:14.973829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:14.973862 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:14.976345 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:14.976994 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:14.977000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:14.977007 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:14.977125 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:14.977467 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:14.977473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:14.977477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:14.977744 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:14.978000 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:14.978270 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 20:52:14.978490 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":313,"threads_started":1}
I20260926 20:52:14.982446 10930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:52:14.983096 10930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:52:14.989711 11068 tablet_service.cc:1467] Processing CreateTablet for tablet a75d423cc9ca41d686000209f83d9abb (DEFAULT_TABLE table=test-table [id=64952437ce4e4bc8909a4042b6d0ccbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:14.989889 11207 tablet_service.cc:1467] Processing CreateTablet for tablet a75d423cc9ca41d686000209f83d9abb (DEFAULT_TABLE table=test-table [id=64952437ce4e4bc8909a4042b6d0ccbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:14.990177 11068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a75d423cc9ca41d686000209f83d9abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:14.990155 11347 tablet_service.cc:1467] Processing CreateTablet for tablet a75d423cc9ca41d686000209f83d9abb (DEFAULT_TABLE table=test-table [id=64952437ce4e4bc8909a4042b6d0ccbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:14.990334 11207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a75d423cc9ca41d686000209f83d9abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:14.990540 11347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a75d423cc9ca41d686000209f83d9abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:14.995090 11539 tablet_bootstrap.cc:492] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe: Bootstrap starting.
I20260926 20:52:14.996556 11539 tablet_bootstrap.cc:654] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:14.998111 11542 tablet_bootstrap.cc:492] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32: Bootstrap starting.
I20260926 20:52:14.998965 11540 tablet_bootstrap.cc:492] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc: Bootstrap starting.
I20260926 20:52:14.999492 11539 tablet_bootstrap.cc:492] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe: No bootstrap required, opened a new log
I20260926 20:52:14.999655 11539 ts_tablet_manager.cc:1397] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260926 20:52:14.999925 11542 tablet_bootstrap.cc:654] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:15.000344 11539 raft_consensus.cc:348] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.000550 11539 raft_consensus.cc:374] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:15.000619 11540 tablet_bootstrap.cc:654] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:15.000635 11539 raft_consensus.cc:729] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbe00975fc74ad6b99f2b4140611cfe, State: Initialized, Role: FOLLOWER
I20260926 20:52:15.000891 11539 consensus_queue.cc:260] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.001533 11539 ts_tablet_manager.cc:1428] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 20:52:15.002701 11542 tablet_bootstrap.cc:492] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32: No bootstrap required, opened a new log
I20260926 20:52:15.002878 11542 ts_tablet_manager.cc:1397] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:52:15.003012 11540 tablet_bootstrap.cc:492] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc: No bootstrap required, opened a new log
I20260926 20:52:15.003162 11540 ts_tablet_manager.cc:1397] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 20:52:15.004006 11540 raft_consensus.cc:348] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.004287 11540 raft_consensus.cc:374] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:15.004385 11540 raft_consensus.cc:729] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99df0a67d849469fa222ca16b8034ffc, State: Initialized, Role: FOLLOWER
I20260926 20:52:15.004659 11540 consensus_queue.cc:260] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.004868 11542 raft_consensus.cc:348] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.005088 11542 raft_consensus.cc:374] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:15.005172 11540 ts_tablet_manager.cc:1428] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:52:15.005188 11542 raft_consensus.cc:729] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94682c641b8b457cbe5552c065e73a32, State: Initialized, Role: FOLLOWER
I20260926 20:52:15.005458 11542 consensus_queue.cc:260] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.006309 11542 ts_tablet_manager.cc:1428] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:52:15.300028 11506 raft_consensus.cc:482] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:15.300223 11506 raft_consensus.cc:504] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.300967 11506 leader_election.cc:288] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94682c641b8b457cbe5552c065e73a32 (127.8.249.130:36381), 99df0a67d849469fa222ca16b8034ffc (127.8.249.131:46559)
I20260926 20:52:15.301550 11372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a75d423cc9ca41d686000209f83d9abb" candidate_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99df0a67d849469fa222ca16b8034ffc" is_pre_election: true
I20260926 20:52:15.301507 11231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a75d423cc9ca41d686000209f83d9abb" candidate_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94682c641b8b457cbe5552c065e73a32" is_pre_election: true
I20260926 20:52:15.301784 11372 raft_consensus.cc:2454] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bbe00975fc74ad6b99f2b4140611cfe in term 0.
I20260926 20:52:15.301800 11231 raft_consensus.cc:2454] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bbe00975fc74ad6b99f2b4140611cfe in term 0.
I20260926 20:52:15.302276 11019 leader_election.cc:302] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bbe00975fc74ad6b99f2b4140611cfe, 94682c641b8b457cbe5552c065e73a32; no voters:
I20260926 20:52:15.302558 11506 raft_consensus.cc:2781] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:52:15.302659 11506 raft_consensus.cc:482] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:15.302785 11506 raft_consensus.cc:3037] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:15.304847 11506 raft_consensus.cc:504] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.305418 11506 leader_election.cc:288] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [CANDIDATE]: Term 1 election: Requested vote from peers 94682c641b8b457cbe5552c065e73a32 (127.8.249.130:36381), 99df0a67d849469fa222ca16b8034ffc (127.8.249.131:46559)
I20260926 20:52:15.305912 11231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a75d423cc9ca41d686000209f83d9abb" candidate_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94682c641b8b457cbe5552c065e73a32"
I20260926 20:52:15.305930 11372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a75d423cc9ca41d686000209f83d9abb" candidate_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99df0a67d849469fa222ca16b8034ffc"
I20260926 20:52:15.306124 11372 raft_consensus.cc:3037] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:15.306142 11231 raft_consensus.cc:3037] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:15.309078 11372 raft_consensus.cc:2454] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bbe00975fc74ad6b99f2b4140611cfe in term 1.
I20260926 20:52:15.309072 11231 raft_consensus.cc:2454] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bbe00975fc74ad6b99f2b4140611cfe in term 1.
I20260926 20:52:15.309513 11022 leader_election.cc:302] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2bbe00975fc74ad6b99f2b4140611cfe, 94682c641b8b457cbe5552c065e73a32, 99df0a67d849469fa222ca16b8034ffc; no voters:
I20260926 20:52:15.309794 11506 raft_consensus.cc:2781] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:15.309934 11506 raft_consensus.cc:686] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 LEADER]: Becoming Leader. State: Replica: 2bbe00975fc74ad6b99f2b4140611cfe, State: Running, Role: LEADER
I20260926 20:52:15.310220 11506 consensus_queue.cc:237] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.312674 10931 catalog_manager.cc:5495] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bbe00975fc74ad6b99f2b4140611cfe (127.8.249.129). New cstate: current_term: 1 leader_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } health_report { overall_health: UNKNOWN } } }
I20260926 20:52:15.382263 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:15.375812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.8.249.129:42499)
0926 20:52:15.376407 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:15.376412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:15.376458 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:52:15.376520 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:15.376525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:15.376624 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:15.376743 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:15.377218 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:15.378617 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:15.380682 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:15.381093 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:15.381187 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:15.381552 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:15.381656 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:15.381723 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:15.381840 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 20:52:15.381898 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":351,"threads_started":1}
I20260926 20:52:15.395099 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:15.390315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:47271 (local address 127.8.249.129:42499)
0926 20:52:15.390535 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:15.390541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:15.390752 (+ 211us) server_negotiation.cc:407] Connection header received
0926 20:52:15.390980 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:15.390984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:15.391046 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:15.391132 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:15.391532 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:15.392222 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:15.393511 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:15.393881 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:15.394005 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:15.394343 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:15.394431 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:15.394528 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:15.394625 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 20:52:15.394738 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":62}
I20260926 20:52:15.395195 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:15.390114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:42499 (local address 127.8.249.129:47271)
0926 20:52:15.390664 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:15.390678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:15.390857 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:15.391150 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:15.391154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:15.391164 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:52:15.391380 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:15.391388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:15.392345 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:15.392351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:15.393377 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:15.393387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:15.394034 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:15.394039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:15.394170 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:15.394200 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:52:15.394526 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:52:15.394537 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 20:52:15.394611 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:52:15.394785 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":30,"thread_start_us":326,"threads_started":1}
I20260926 20:52:15.402422 11372 raft_consensus.cc:1261] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Refusing update from remote peer 2bbe00975fc74ad6b99f2b4140611cfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:15.402540 11231 raft_consensus.cc:1261] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Refusing update from remote peer 2bbe00975fc74ad6b99f2b4140611cfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:15.402968 11506 consensus_queue.cc:1035] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:15.403293 11523 consensus_queue.cc:1035] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:15.460166 10931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:15.460469 10931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:15.463857 10931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4: Sending DeleteTablet for 3 replicas of tablet a75d423cc9ca41d686000209f83d9abb
I20260926 20:52:15.464823 11207 tablet_service.cc:1514] Processing DeleteTablet for tablet a75d423cc9ca41d686000209f83d9abb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:15 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:60520
I20260926 20:52:15.464825 11347 tablet_service.cc:1514] Processing DeleteTablet for tablet a75d423cc9ca41d686000209f83d9abb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:15 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:51548
I20260926 20:52:15.465468 11068 tablet_service.cc:1514] Processing DeleteTablet for tablet a75d423cc9ca41d686000209f83d9abb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:15 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:53426
I20260926 20:52:15.465816 10931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:52:15.466611 10931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:52:15.469157 11557 tablet_replica.cc:330] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32: stopping tablet replica
I20260926 20:52:15.469375 11557 raft_consensus.cc:2229] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:15.469630 11557 raft_consensus.cc:2258] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:15.471150 11557 ts_tablet_manager.cc:1905] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:15.476876 11346 tablet_service.cc:1467] Processing CreateTablet for tablet 266e4b24217a495d89af0f41b3664bfd (DEFAULT_TABLE table=test-table [id=d0a0c6797256438fa72e944c959aea0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:15.477352 11346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266e4b24217a495d89af0f41b3664bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:15.479820 11557 ts_tablet_manager.cc:1918] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:52:15.479959 11557 log.cc:1196] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/a75d423cc9ca41d686000209f83d9abb
I20260926 20:52:15.480327 11558 tablet_replica.cc:330] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe: stopping tablet replica
I20260926 20:52:15.480404 11557 ts_tablet_manager.cc:1939] T a75d423cc9ca41d686000209f83d9abb P 94682c641b8b457cbe5552c065e73a32: Deleting consensus metadata
I20260926 20:52:15.480666 11558 raft_consensus.cc:2229] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:52:15.481352 11558 raft_consensus.cc:2258] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:15.482416 11207 tablet_service.cc:1467] Processing CreateTablet for tablet 266e4b24217a495d89af0f41b3664bfd (DEFAULT_TABLE table=test-table [id=d0a0c6797256438fa72e944c959aea0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:15.482530 11558 ts_tablet_manager.cc:1905] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:15.482883 11207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266e4b24217a495d89af0f41b3664bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:15.482889 10916 catalog_manager.cc:4841] TS 94682c641b8b457cbe5552c065e73a32 (127.8.249.130:36381): tablet a75d423cc9ca41d686000209f83d9abb (table test-table [id=64952437ce4e4bc8909a4042b6d0ccbb]) successfully deleted
I20260926 20:52:15.489041 11542 tablet_bootstrap.cc:492] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32: Bootstrap starting.
I20260926 20:52:15.489717 11068 tablet_service.cc:1467] Processing CreateTablet for tablet 266e4b24217a495d89af0f41b3664bfd (DEFAULT_TABLE table=test-table [id=d0a0c6797256438fa72e944c959aea0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:15.490029 11558 ts_tablet_manager.cc:1918] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:52:15.490172 11068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266e4b24217a495d89af0f41b3664bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:15.490219 11558 log.cc:1196] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/a75d423cc9ca41d686000209f83d9abb
I20260926 20:52:15.490825 11542 tablet_bootstrap.cc:654] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:15.490832 11558 ts_tablet_manager.cc:1939] T a75d423cc9ca41d686000209f83d9abb P 2bbe00975fc74ad6b99f2b4140611cfe: Deleting consensus metadata
I20260926 20:52:15.492098 10916 catalog_manager.cc:4841] TS 2bbe00975fc74ad6b99f2b4140611cfe (127.8.249.129:42499): tablet a75d423cc9ca41d686000209f83d9abb (table test-table [id=64952437ce4e4bc8909a4042b6d0ccbb]) successfully deleted
I20260926 20:52:15.494613 11540 tablet_bootstrap.cc:492] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc: Bootstrap starting.
I20260926 20:52:15.496155 11540 tablet_bootstrap.cc:654] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:15.504467 11542 tablet_bootstrap.cc:492] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32: No bootstrap required, opened a new log
I20260926 20:52:15.504691 11542 ts_tablet_manager.cc:1397] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20260926 20:52:15.505700 11542 raft_consensus.cc:348] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.505992 11542 raft_consensus.cc:374] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:15.506073 11542 raft_consensus.cc:729] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94682c641b8b457cbe5552c065e73a32, State: Initialized, Role: FOLLOWER
I20260926 20:52:15.506348 11542 consensus_queue.cc:260] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.507838 11540 tablet_bootstrap.cc:492] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc: No bootstrap required, opened a new log
I20260926 20:52:15.508021 11540 ts_tablet_manager.cc:1397] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260926 20:52:15.508793 11540 raft_consensus.cc:348] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.509053 11540 raft_consensus.cc:374] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:15.509138 11540 raft_consensus.cc:729] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99df0a67d849469fa222ca16b8034ffc, State: Initialized, Role: FOLLOWER
I20260926 20:52:15.509326 11542 ts_tablet_manager.cc:1428] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260926 20:52:15.509438 11540 consensus_queue.cc:260] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.509987 11540 ts_tablet_manager.cc:1428] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 20:52:15.510703 11561 tablet_bootstrap.cc:492] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe: Bootstrap starting.
I20260926 20:52:15.512600 11561 tablet_bootstrap.cc:654] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:15.515218 11556 tablet_replica.cc:330] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc: stopping tablet replica
I20260926 20:52:15.515458 11556 raft_consensus.cc:2229] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:15.515671 11556 raft_consensus.cc:2258] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:15.516768 11556 ts_tablet_manager.cc:1905] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:15.524348 11556 ts_tablet_manager.cc:1918] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:52:15.524629 11556 log.cc:1196] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/a75d423cc9ca41d686000209f83d9abb
I20260926 20:52:15.525147 11556 ts_tablet_manager.cc:1939] T a75d423cc9ca41d686000209f83d9abb P 99df0a67d849469fa222ca16b8034ffc: Deleting consensus metadata
I20260926 20:52:15.526384 10919 catalog_manager.cc:4841] TS 99df0a67d849469fa222ca16b8034ffc (127.8.249.131:46559): tablet a75d423cc9ca41d686000209f83d9abb (table test-table [id=64952437ce4e4bc8909a4042b6d0ccbb]) successfully deleted
I20260926 20:52:15.529596 11561 tablet_bootstrap.cc:492] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe: No bootstrap required, opened a new log
I20260926 20:52:15.529764 11561 ts_tablet_manager.cc:1397] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.009s
I20260926 20:52:15.530469 11561 raft_consensus.cc:348] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.530658 11561 raft_consensus.cc:374] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:15.530781 11561 raft_consensus.cc:729] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbe00975fc74ad6b99f2b4140611cfe, State: Initialized, Role: FOLLOWER
I20260926 20:52:15.531069 11561 consensus_queue.cc:260] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.531559 11561 ts_tablet_manager.cc:1428] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:52:15.576197 11545 raft_consensus.cc:482] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:15.576397 11545 raft_consensus.cc:504] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.577159 11545 leader_election.cc:288] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bbe00975fc74ad6b99f2b4140611cfe (127.8.249.129:42499), 99df0a67d849469fa222ca16b8034ffc (127.8.249.131:46559)
I20260926 20:52:15.577761 11372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266e4b24217a495d89af0f41b3664bfd" candidate_uuid: "94682c641b8b457cbe5552c065e73a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99df0a67d849469fa222ca16b8034ffc" is_pre_election: true
I20260926 20:52:15.577780 11094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266e4b24217a495d89af0f41b3664bfd" candidate_uuid: "94682c641b8b457cbe5552c065e73a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" is_pre_election: true
I20260926 20:52:15.578022 11372 raft_consensus.cc:2454] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94682c641b8b457cbe5552c065e73a32 in term 0.
I20260926 20:52:15.578078 11094 raft_consensus.cc:2454] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94682c641b8b457cbe5552c065e73a32 in term 0.
I20260926 20:52:15.578503 11162 leader_election.cc:302] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94682c641b8b457cbe5552c065e73a32, 99df0a67d849469fa222ca16b8034ffc; no voters:
I20260926 20:52:15.578893 11545 raft_consensus.cc:2781] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:52:15.579030 11545 raft_consensus.cc:482] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:15.579129 11545 raft_consensus.cc:3037] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:15.581166 11545 raft_consensus.cc:504] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.581825 11545 leader_election.cc:288] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [CANDIDATE]: Term 1 election: Requested vote from peers 2bbe00975fc74ad6b99f2b4140611cfe (127.8.249.129:42499), 99df0a67d849469fa222ca16b8034ffc (127.8.249.131:46559)
I20260926 20:52:15.582417 11094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266e4b24217a495d89af0f41b3664bfd" candidate_uuid: "94682c641b8b457cbe5552c065e73a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bbe00975fc74ad6b99f2b4140611cfe"
I20260926 20:52:15.582468 11372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266e4b24217a495d89af0f41b3664bfd" candidate_uuid: "94682c641b8b457cbe5552c065e73a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99df0a67d849469fa222ca16b8034ffc"
I20260926 20:52:15.582667 11094 raft_consensus.cc:3037] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:15.582674 11372 raft_consensus.cc:3037] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:15.585711 11372 raft_consensus.cc:2454] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94682c641b8b457cbe5552c065e73a32 in term 1.
I20260926 20:52:15.585743 11094 raft_consensus.cc:2454] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94682c641b8b457cbe5552c065e73a32 in term 1.
I20260926 20:52:15.586222 11159 leader_election.cc:302] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bbe00975fc74ad6b99f2b4140611cfe, 94682c641b8b457cbe5552c065e73a32; no voters:
I20260926 20:52:15.586567 11545 raft_consensus.cc:2781] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:15.587119 11545 raft_consensus.cc:686] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [term 1 LEADER]: Becoming Leader. State: Replica: 94682c641b8b457cbe5552c065e73a32, State: Running, Role: LEADER
I20260926 20:52:15.587734 11545 consensus_queue.cc:237] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } }
I20260926 20:52:15.590588 10931 catalog_manager.cc:5495] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94682c641b8b457cbe5552c065e73a32 (127.8.249.130). New cstate: current_term: 1 leader_uuid: "94682c641b8b457cbe5552c065e73a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94682c641b8b457cbe5552c065e73a32" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 } health_report { overall_health: UNKNOWN } } }
I20260926 20:52:15.630156 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:15.624420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.8.249.130:36381)
0926 20:52:15.624974 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:15.624980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:15.624999 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:52:15.625047 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:15.625050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:15.625105 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:15.625183 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:15.625620 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:15.626345 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:15.628264 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:15.628664 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:15.628751 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:15.629273 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:15.629374 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:15.629438 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:15.629709 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 20:52:15.629804 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":318,"threads_started":1}
I20260926 20:52:15.639763 11094 raft_consensus.cc:1261] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Refusing update from remote peer 94682c641b8b457cbe5552c065e73a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:15.639896 11372 raft_consensus.cc:1261] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Refusing update from remote peer 94682c641b8b457cbe5552c065e73a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:15.640409 11563 consensus_queue.cc:1035] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bbe00975fc74ad6b99f2b4140611cfe" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:15.640728 11545 consensus_queue.cc:1035] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99df0a67d849469fa222ca16b8034ffc" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:52:15.667093 11511 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:52:17.255434 10932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:17.255640 10932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:17.259366 10932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab697a91e2124def80684f69c02e7fc4: Sending DeleteTablet for 3 replicas of tablet 266e4b24217a495d89af0f41b3664bfd
I20260926 20:52:17.260128 11068 tablet_service.cc:1514] Processing DeleteTablet for tablet 266e4b24217a495d89af0f41b3664bfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:17 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:53426
I20260926 20:52:17.260144 11207 tablet_service.cc:1514] Processing DeleteTablet for tablet 266e4b24217a495d89af0f41b3664bfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:17 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:60520
I20260926 20:52:17.260381 11347 tablet_service.cc:1514] Processing DeleteTablet for tablet 266e4b24217a495d89af0f41b3664bfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:17 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:51548
I20260926 20:52:17.261471 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 11003
I20260926 20:52:17.263713 11596 tablet_replica.cc:330] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc: stopping tablet replica
I20260926 20:52:17.264046 11596 raft_consensus.cc:2229] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:17.264443 11596 raft_consensus.cc:2258] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:17.266247 11596 ts_tablet_manager.cc:1905] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:17.275334 11596 ts_tablet_manager.cc:1918] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:52:17.275604 11596 log.cc:1196] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/266e4b24217a495d89af0f41b3664bfd
I20260926 20:52:17.276280 11596 ts_tablet_manager.cc:1939] T 266e4b24217a495d89af0f41b3664bfd P 99df0a67d849469fa222ca16b8034ffc: Deleting consensus metadata
I20260926 20:52:17.277777 10919 catalog_manager.cc:4841] TS 99df0a67d849469fa222ca16b8034ffc (127.8.249.131:46559): tablet 266e4b24217a495d89af0f41b3664bfd (table test-table [id=d0a0c6797256438fa72e944c959aea0e]) successfully deleted
I20260926 20:52:17.279202 11595 tablet_replica.cc:330] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe: stopping tablet replica
I20260926 20:52:17.279433 11595 raft_consensus.cc:2229] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:17.279724 11595 raft_consensus.cc:2258] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:17.279929 11594 tablet_replica.cc:330] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32: stopping tablet replica
I20260926 20:52:17.280149 11594 raft_consensus.cc:2229] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:52:17.280539 11594 raft_consensus.cc:2258] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:17.280643 11595 ts_tablet_manager.cc:1905] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:17.281383 11594 ts_tablet_manager.cc:1905] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:17.287421 11595 ts_tablet_manager.cc:1918] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:52:17.287629 11595 log.cc:1196] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/266e4b24217a495d89af0f41b3664bfd
I20260926 20:52:17.287896 11594 ts_tablet_manager.cc:1918] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:52:17.288043 11594 log.cc:1196] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/266e4b24217a495d89af0f41b3664bfd
I20260926 20:52:17.288106 11595 ts_tablet_manager.cc:1939] T 266e4b24217a495d89af0f41b3664bfd P 2bbe00975fc74ad6b99f2b4140611cfe: Deleting consensus metadata
I20260926 20:52:17.288451 11594 ts_tablet_manager.cc:1939] T 266e4b24217a495d89af0f41b3664bfd P 94682c641b8b457cbe5552c065e73a32: Deleting consensus metadata
I20260926 20:52:17.289327 10916 catalog_manager.cc:4841] TS 2bbe00975fc74ad6b99f2b4140611cfe (127.8.249.129:42499): tablet 266e4b24217a495d89af0f41b3664bfd (table test-table [id=d0a0c6797256438fa72e944c959aea0e]) successfully deleted
I20260926 20:52:17.289572 10916 catalog_manager.cc:4841] TS 94682c641b8b457cbe5552c065e73a32 (127.8.249.130:36381): tablet 266e4b24217a495d89af0f41b3664bfd (table test-table [id=d0a0c6797256438fa72e944c959aea0e]) successfully deleted
Sep 26 20:52:17 dist-test-slave-2dg3 krb5kdc[10887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:52:17.299002 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:17.262911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.8.249.129:42499)
0926 20:52:17.279043 (+ 16132us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:17.279054 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:17.279083 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:52:17.279145 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:17.279149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:17.279233 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:17.279398 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:17.281202 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:17.282071 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:17.283398 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:17.283851 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:17.283996 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:17.293448 (+ 9452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:17.293468 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:17.293485 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:17.293526 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:17.296369 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:17.297139 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:17.297148 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:17.297150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:17.297278 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:17.297661 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:17.297664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:17.297666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:17.298023 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:17.298232 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:17.298500 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 20:52:17.298657 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15953,"thread_start_us":15777,"threads_started":1}
W20260926 20:52:17.299650 11134 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44908
W20260926 20:52:17.300304 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 11003: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:52:17.300535 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 11003
I20260926 20:52:17.326445 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 11143
Sep 26 20:52:17 dist-test-slave-2dg3 krb5kdc[10887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:52:17.341590 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:17.327207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43740 (local address 127.8.249.130:36381)
0926 20:52:17.327781 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:17.327786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:17.327802 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:52:17.327854 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:17.327858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:17.327915 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:17.328044 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:17.329261 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:17.330003 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:17.331317 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:17.331731 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:17.331863 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:17.336137 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:17.336150 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:17.336165 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:17.336210 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:17.339065 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:17.339744 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:17.339752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:17.339754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:17.339881 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:17.340213 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:17.340216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:17.340218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:17.340571 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:17.340785 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:17.341017 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 20:52:17.341192 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":289,"threads_started":1}
W20260926 20:52:17.342356 11273 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43740
W20260926 20:52:17.343019 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 11143: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:52:17.343175 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 11143
I20260926 20:52:17.365315 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 11282
Sep 26 20:52:17 dist-test-slave-2dg3 krb5kdc[10887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:52:17.379654 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:17.366068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.8.249.131:46559)
0926 20:52:17.366556 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:17.366560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:17.366577 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:52:17.366630 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:17.366634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:17.366693 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:17.366852 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:17.368138 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:17.368881 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:17.370072 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:17.370468 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:17.370574 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:17.374743 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:17.374758 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:17.374774 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:17.374875 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:17.377403 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:17.377965 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:17.377971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:17.377973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:17.378093 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:17.378383 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:17.378386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:17.378388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:17.378717 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:17.378992 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:17.379228 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 20:52:17.379397 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":272,"threads_started":1}
W20260926 20:52:17.380314 11412 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51658
W20260926 20:52:17.380846 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 11282: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:52:17.380996 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 11282
I20260926 20:52:17.402220 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 10900
I20260926 20:52:17.403426 10972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:17.520144 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 10900
2026-09-26T20:52:17Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8715 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:52:17 dist-test-slave-2dg3 krb5kdc[11604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:52:17 dist-test-slave-2dg3 krb5kdc[11604](info): set up 2 sockets
Sep 26 20:52:17 dist-test-slave-2dg3 krb5kdc[11604](info): commencing operation
krb5kdc: starting...
W20260926 20:52:18.603048 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:52:18 dist-test-slave-2dg3 krb5kdc[11604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:52:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:52:18.818037 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:40117
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:40117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:19.146672 11617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:19.147087 11617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:19.147265 11617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:19.154716 11617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:19.154938 11617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:19.155056 11617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:19.155170 11617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:19.155253 11617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:19.165761 11617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:40117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:19.167282 11617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:19.168675 11617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:19.181154 11625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:19.181221 11622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:19.181219 11623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:19.181988 11617 server_base.cc:1034] running on GCE node
I20260926 20:52:19.182761 11617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:19.184352 11617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:19.185576 11617 hybrid_clock.cc:648] HybridClock initialized: now 1790455939185540 us; error 30 us; skew 500 ppm
I20260926 20:52:19.185889 11617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:19 dist-test-slave-2dg3 krb5kdc[11604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455939, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:19.189051 11617 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:52:19.190452 11617 webserver.cc:460] Webserver started at http://127.8.249.190:38397/ using document root <none> and password file <none>
I20260926 20:52:19.191183 11617 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:19.191290 11617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:19.191545 11617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:19.194448 11617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "430c7b5dbe05446fb95a725ac8d2ae82"
format_stamp: "Formatted at 2026-09-26 20:52:19 on dist-test-slave-2dg3"
server_key: "0ac877cc3db12711013c6c6b0de61662"
server_key_iv: "2c58427e9dc21010d2a72cc29fa494d6"
server_key_version: "encryptionkey@0"
I20260926 20:52:19.195227 11617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "430c7b5dbe05446fb95a725ac8d2ae82"
format_stamp: "Formatted at 2026-09-26 20:52:19 on dist-test-slave-2dg3"
server_key: "0ac877cc3db12711013c6c6b0de61662"
server_key_iv: "2c58427e9dc21010d2a72cc29fa494d6"
server_key_version: "encryptionkey@0"
I20260926 20:52:19.200533 11617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:52:19.204162 11632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:19.205557 11617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:52:19.205708 11617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "430c7b5dbe05446fb95a725ac8d2ae82"
format_stamp: "Formatted at 2026-09-26 20:52:19 on dist-test-slave-2dg3"
server_key: "0ac877cc3db12711013c6c6b0de61662"
server_key_iv: "2c58427e9dc21010d2a72cc29fa494d6"
server_key_version: "encryptionkey@0"
I20260926 20:52:19.205879 11617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:19.229357 11617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:19.233726 11617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:19.233964 11617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:19.252523 11617 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:40117
I20260926 20:52:19.252530 11683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:40117 every 8 connection(s)
I20260926 20:52:19.253949 11617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:19.257099 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 11617
I20260926 20:52:19.257341 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:19.258117 9190 external_mini_cluster.cc:1428] Setting key 20e25de6179b0d3b2b16464127cc3c48
I20260926 20:52:19.258342 11684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:19.273372 11684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82: Bootstrap starting.
Sep 26 20:52:19 dist-test-slave-2dg3 krb5kdc[11604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:19.276866 11684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:19.277925 11684 log.cc:824] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:19.280753 11684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82: No bootstrap required, opened a new log
I20260926 20:52:19.281810 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:19.260309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.8.249.190:40117)
0926 20:52:19.261148 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:19.261160 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:19.261207 (+ 47us) server_negotiation.cc:407] Connection header received
0926 20:52:19.263299 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:19.263323 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:19.263768 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:19.264153 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:19.265995 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:19.267394 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:19.269132 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:19.269836 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:19.269974 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:19.274604 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:19.274612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:19.274631 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:19.274671 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:19.277781 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:19.278523 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:19.278538 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:19.278544 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:19.278718 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:19.279128 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:19.279135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:19.279138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:19.279640 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:19.279929 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:19.280367 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 20:52:19.280781 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":387,"threads_started":1}
I20260926 20:52:19.288875 11684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430c7b5dbe05446fb95a725ac8d2ae82" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40117 } }
I20260926 20:52:19.289330 11684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:19.289431 11684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430c7b5dbe05446fb95a725ac8d2ae82, State: Initialized, Role: FOLLOWER
I20260926 20:52:19.289988 11684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430c7b5dbe05446fb95a725ac8d2ae82" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40117 } }
I20260926 20:52:19.290189 11684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:19.290272 11684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:19.290390 11684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:19.292534 11684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430c7b5dbe05446fb95a725ac8d2ae82" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40117 } }
I20260926 20:52:19.292986 11684 leader_election.cc:302] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 430c7b5dbe05446fb95a725ac8d2ae82; no voters:
I20260926 20:52:19.293532 11684 leader_election.cc:288] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:19.293681 11689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:19.294409 11689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [term 1 LEADER]: Becoming Leader. State: Replica: 430c7b5dbe05446fb95a725ac8d2ae82, State: Running, Role: LEADER
I20260926 20:52:19.295068 11689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430c7b5dbe05446fb95a725ac8d2ae82" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40117 } }
I20260926 20:52:19.295780 11684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:19.298236 11690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "430c7b5dbe05446fb95a725ac8d2ae82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430c7b5dbe05446fb95a725ac8d2ae82" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40117 } } }
I20260926 20:52:19.298552 11690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:19.298975 11691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 430c7b5dbe05446fb95a725ac8d2ae82. Latest consensus state: current_term: 1 leader_uuid: "430c7b5dbe05446fb95a725ac8d2ae82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "430c7b5dbe05446fb95a725ac8d2ae82" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40117 } } }
I20260926 20:52:19.299417 11691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:19.300915 11696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:19.307552 11696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:19.316495 11696 catalog_manager.cc:1347] Generated new cluster ID: 912acd7339d147f68d85ab6c233b0ce1
I20260926 20:52:19.316633 11696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:19.342430 11696 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:19.343961 11696 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:19.357195 11696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 430c7b5dbe05446fb95a725ac8d2ae82: Generated new TSK 0
I20260926 20:52:19.358122 11696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:52:19.416687 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40117
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:19.793912 11710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:19.794346 11710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:19.794560 11710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:19.810230 11710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:19.810406 11710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:19.810683 11710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:52:19.832146 11710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:19.834187 11710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:19.836125 11710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:19.853199 11716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:19.853325 11718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:19.854036 11715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:19.854414 11710 server_base.cc:1034] running on GCE node
I20260926 20:52:19.855316 11710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:19.856813 11710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:19.858192 11710 hybrid_clock.cc:648] HybridClock initialized: now 1790455939858084 us; error 92 us; skew 500 ppm
I20260926 20:52:19.858669 11710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:19 dist-test-slave-2dg3 krb5kdc[11604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455939, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:19.863062 11710 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:52:19.864671 11710 webserver.cc:460] Webserver started at http://127.8.249.129:38687/ using document root <none> and password file <none>
I20260926 20:52:19.865389 11710 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:19.865533 11710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:19.865844 11710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:19.868980 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "8cb73097731f4a86b81910dce49da63f"
format_stamp: "Formatted at 2026-09-26 20:52:19 on dist-test-slave-2dg3"
server_key: "5a7f9ef29084ed0862fe48169099889a"
server_key_iv: "325ef354c4543c821f1fe10e4b7413f3"
server_key_version: "encryptionkey@0"
I20260926 20:52:19.869813 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "8cb73097731f4a86b81910dce49da63f"
format_stamp: "Formatted at 2026-09-26 20:52:19 on dist-test-slave-2dg3"
server_key: "5a7f9ef29084ed0862fe48169099889a"
server_key_iv: "325ef354c4543c821f1fe10e4b7413f3"
server_key_version: "encryptionkey@0"
I20260926 20:52:19.875599 11710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 20:52:19.879622 11725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:19.881194 11710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:52:19.881392 11710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "8cb73097731f4a86b81910dce49da63f"
format_stamp: "Formatted at 2026-09-26 20:52:19 on dist-test-slave-2dg3"
server_key: "5a7f9ef29084ed0862fe48169099889a"
server_key_iv: "325ef354c4543c821f1fe10e4b7413f3"
server_key_version: "encryptionkey@0"
I20260926 20:52:19.881608 11710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:19.897047 11710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:19.901667 11710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:19.901968 11710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:19.902992 11710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:19.904429 11710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:19.904526 11710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:19.904716 11710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:19.904804 11710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:19.947429 11710 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:38749
I20260926 20:52:19.947470 11837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:38749 every 8 connection(s)
I20260926 20:52:19.948717 11710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:52:19.949999 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 11710
I20260926 20:52:19.950258 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:52:19.951231 9190 external_mini_cluster.cc:1428] Setting key 7055b4d8baaec72248d4623cbab3a2b0
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
Sep 26 20:52:19 dist-test-slave-2dg3 krb5kdc[11604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455939, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:19.989773 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:19.953412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:35051 (local address 127.8.249.190:40117)
0926 20:52:19.961323 (+ 7911us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:19.961332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:19.966968 (+ 5636us) server_negotiation.cc:407] Connection header received
0926 20:52:19.971793 (+ 4825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:19.971805 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:19.971920 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:19.972009 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:19.973891 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:19.974627 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:19.976325 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:19.976805 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:19.976884 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:19.982372 (+ 5488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:19.982382 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:19.982389 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:19.982440 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:19.986482 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:19.987321 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:19.987325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:19.987328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:19.987523 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:19.987993 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:19.987998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:19.988001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:19.988283 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:19.988470 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:19.989176 (+ 706us) server_negotiation.cc:299] Negotiation successful
0926 20:52:19.989399 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7748,"thread_start_us":7602,"threads_started":1}
I20260926 20:52:19.990868 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:19.952773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40117 (local address 127.8.249.129:35051)
0926 20:52:19.966726 (+ 13953us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:19.966807 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:19.971356 (+ 4549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:19.972579 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:19.972599 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:19.973127 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:19.973704 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:19.973723 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:19.974817 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:19.974823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:19.976154 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:19.976172 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:19.977173 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:19.977178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:19.977376 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:19.978126 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:19.978150 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:19.982069 (+ 3919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:19.986631 (+ 4562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:19.986643 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:19.986664 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:19.987174 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:19.987663 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:19.987667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:19.987670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:19.987864 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:19.988412 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:19.988423 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:19.988790 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 20:52:19.989065 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 20:52:19.989600 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13505,"thread_start_us":14081,"threads_started":1}
I20260926 20:52:19.992489 11838 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40117
I20260926 20:52:19.992862 11838 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:19.993821 11838 heartbeater.cc:511] Master 127.8.249.190:40117 requested a full tablet report, sending...
I20260926 20:52:19.996227 11649 ts_manager.cc:194] Registered new tserver with Master: 8cb73097731f4a86b81910dce49da63f (127.8.249.129:38749)
I20260926 20:52:19.997742 11649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:35051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:52:20.007031 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40117
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:20.331511 11844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:20.331863 11844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:20.332041 11844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:20.339407 11844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:20.339591 11844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:20.339772 11844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:52:20.349800 11844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:20.351199 11844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:20.352553 11844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:20.365919 11849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:20.365948 11852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:20.365880 11850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:20.366786 11844 server_base.cc:1034] running on GCE node
I20260926 20:52:20.367614 11844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:20.368671 11844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:20.369930 11844 hybrid_clock.cc:648] HybridClock initialized: now 1790455940369854 us; error 65 us; skew 500 ppm
I20260926 20:52:20.370299 11844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:20 dist-test-slave-2dg3 krb5kdc[11604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455940, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:20.373611 11844 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:52:20.375490 11844 webserver.cc:460] Webserver started at http://127.8.249.130:46405/ using document root <none> and password file <none>
I20260926 20:52:20.376339 11844 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:20.376497 11844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:20.376837 11844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:20.379973 11844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "3903f48cd50041b09ab0579967d1ce5d"
format_stamp: "Formatted at 2026-09-26 20:52:20 on dist-test-slave-2dg3"
server_key: "cdbfe65fb10860d3283f682cdef37b88"
server_key_iv: "06e3ffee4958b893709fe5131b968a3e"
server_key_version: "encryptionkey@0"
I20260926 20:52:20.380831 11844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "3903f48cd50041b09ab0579967d1ce5d"
format_stamp: "Formatted at 2026-09-26 20:52:20 on dist-test-slave-2dg3"
server_key: "cdbfe65fb10860d3283f682cdef37b88"
server_key_iv: "06e3ffee4958b893709fe5131b968a3e"
server_key_version: "encryptionkey@0"
I20260926 20:52:20.386894 11844 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260926 20:52:20.390588 11859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:20.392074 11844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:20.392285 11844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "3903f48cd50041b09ab0579967d1ce5d"
format_stamp: "Formatted at 2026-09-26 20:52:20 on dist-test-slave-2dg3"
server_key: "cdbfe65fb10860d3283f682cdef37b88"
server_key_iv: "06e3ffee4958b893709fe5131b968a3e"
server_key_version: "encryptionkey@0"
I20260926 20:52:20.392518 11844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:20.413576 11844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:20.417904 11844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:20.418253 11844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:20.419319 11844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:20.420826 11844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:20.420939 11844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:20.421108 11844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:20.421187 11844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:20.456447 11844 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:42525
I20260926 20:52:20.456491 11971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:42525 every 8 connection(s)
I20260926 20:52:20.457805 11844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:52:20.467865 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 11844
I20260926 20:52:20.468057 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:52:20.468483 9190 external_mini_cluster.cc:1428] Setting key e795cc759b224af902154206f4d951a2
Sep 26 20:52:20 dist-test-slave-2dg3 krb5kdc[11604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455940, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:20.479378 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:20.460506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:39749 (local address 127.8.249.190:40117)
0926 20:52:20.460707 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:20.460712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:20.461956 (+ 1244us) server_negotiation.cc:407] Connection header received
0926 20:52:20.464033 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:20.464040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:20.464143 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:20.464268 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:20.465884 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:20.466618 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:20.468414 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:20.468983 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:20.469137 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:20.473206 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:20.473212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:20.473215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:20.473251 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:20.476069 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:20.476868 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:20.476874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:20.476876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:20.477010 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:20.477484 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:20.477496 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:20.477500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:20.477779 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:20.477960 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:20.478709 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 20:52:20.478982 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:52:20.480405 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:20.461001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40117 (local address 127.8.249.130:39749)
0926 20:52:20.461754 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:20.461797 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:20.463716 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:20.464536 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:20.464544 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:20.465083 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:20.465679 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:20.465698 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:20.466777 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:20.466787 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:20.468257 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:20.468270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:20.469228 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:20.469235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:20.469546 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:20.470324 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:20.470362 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:20.472946 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:20.476226 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:20.476244 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:20.476264 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:20.476719 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:20.477130 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:20.477137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:20.477141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:20.477356 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:20.477917 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:20.477926 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:20.478305 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 20:52:20.478589 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 20:52:20.479185 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":281,"threads_started":1}
I20260926 20:52:20.482035 11972 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40117
I20260926 20:52:20.482398 11972 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:20.483247 11972 heartbeater.cc:511] Master 127.8.249.190:40117 requested a full tablet report, sending...
I20260926 20:52:20.484788 11649 ts_manager.cc:194] Registered new tserver with Master: 3903f48cd50041b09ab0579967d1ce5d (127.8.249.130:42525)
I20260926 20:52:20.485743 11649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:39749
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:52:20.513715 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40117
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:20.837085 11977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:20.837371 11977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:20.837525 11977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:20.844944 11977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:20.845062 11977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:20.845199 11977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:52:20.855202 11977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:20.856599 11977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:20.857973 11977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:20.871042 11983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:20.871120 11982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:20.871074 11985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:20.871766 11977 server_base.cc:1034] running on GCE node
I20260926 20:52:20.872303 11977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:20.873188 11977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:20.874380 11977 hybrid_clock.cc:648] HybridClock initialized: now 1790455940874327 us; error 42 us; skew 500 ppm
I20260926 20:52:20.874687 11977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:20 dist-test-slave-2dg3 krb5kdc[11604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455940, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:20.877694 11977 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:52:20.879326 11977 webserver.cc:460] Webserver started at http://127.8.249.131:44577/ using document root <none> and password file <none>
I20260926 20:52:20.880013 11977 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:20.880151 11977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:20.880461 11977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:20.883354 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "8b0379f8d13648c0abcf258b08f9f629"
format_stamp: "Formatted at 2026-09-26 20:52:20 on dist-test-slave-2dg3"
server_key: "9897c4b3b7f767464a5c94494c3c6b45"
server_key_iv: "40a28165d40b6edaf8dbeb007a229e8d"
server_key_version: "encryptionkey@0"
I20260926 20:52:20.884114 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "8b0379f8d13648c0abcf258b08f9f629"
format_stamp: "Formatted at 2026-09-26 20:52:20 on dist-test-slave-2dg3"
server_key: "9897c4b3b7f767464a5c94494c3c6b45"
server_key_iv: "40a28165d40b6edaf8dbeb007a229e8d"
server_key_version: "encryptionkey@0"
I20260926 20:52:20.889413 11977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 20:52:20.892791 11992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:20.894114 11977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:52:20.894289 11977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "8b0379f8d13648c0abcf258b08f9f629"
format_stamp: "Formatted at 2026-09-26 20:52:20 on dist-test-slave-2dg3"
server_key: "9897c4b3b7f767464a5c94494c3c6b45"
server_key_iv: "40a28165d40b6edaf8dbeb007a229e8d"
server_key_version: "encryptionkey@0"
I20260926 20:52:20.894484 11977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:20.917011 11977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:20.921305 11977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:20.921631 11977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:20.922691 11977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:20.924170 11977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:20.924291 11977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:20.924450 11977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:20.924532 11977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:20.960146 11977 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:40127
I20260926 20:52:20.960184 12104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:40127 every 8 connection(s)
I20260926 20:52:20.961566 11977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:52:20.963339 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 11977
I20260926 20:52:20.963538 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:52:20.964017 9190 external_mini_cluster.cc:1428] Setting key b2bdee999ddd4d6c6076be636616416f
Sep 26 20:52:20 dist-test-slave-2dg3 krb5kdc[11604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455940, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:20.982352 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:20.964518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:51781 (local address 127.8.249.190:40117)
0926 20:52:20.964719 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:20.964723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:20.966153 (+ 1430us) server_negotiation.cc:407] Connection header received
0926 20:52:20.968294 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:20.968299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:20.968387 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:20.968480 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:20.969952 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:20.970679 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:20.972386 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:20.972768 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:20.972871 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:20.976566 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:20.976572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:20.976575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:20.976611 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:20.979606 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:20.980417 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:20.980421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:20.980423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:20.980535 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:20.980928 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:20.980931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:20.980933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:20.981192 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:20.981302 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:20.981950 (+ 648us) server_negotiation.cc:299] Negotiation successful
0926 20:52:20.982105 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 20:52:20.983386 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:20.965078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40117 (local address 127.8.249.131:51781)
0926 20:52:20.965995 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:20.966039 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:20.968039 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:20.968683 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:20.968692 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:20.969204 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:20.969775 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:20.969796 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:20.970943 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:20.970947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:20.972254 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:20.972267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:20.972990 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:20.972995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:20.973211 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:20.973962 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:20.973985 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:20.976323 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:20.979763 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:20.979770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:20.979785 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:20.980303 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:20.980625 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:20.980628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:20.980630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:20.980801 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:20.981298 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:20.981303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:20.981636 (+ 333us) client_negotiation.cc:769] Sending connection context
0926 20:52:20.981859 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 20:52:20.982247 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":381,"threads_started":1}
I20260926 20:52:20.984596 12105 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40117
I20260926 20:52:20.984894 12105 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:20.985638 12105 heartbeater.cc:511] Master 127.8.249.190:40117 requested a full tablet report, sending...
I20260926 20:52:20.987020 11649 ts_manager.cc:194] Registered new tserver with Master: 8b0379f8d13648c0abcf258b08f9f629 (127.8.249.131:40127)
I20260926 20:52:20.987814 11649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:51781
I20260926 20:52:20.991976 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:52:21.001160 11838 heartbeater.cc:503] Master 127.8.249.190:40117 was elected leader, sending a full tablet report...
W20260926 20:52:21.023859 11841 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:52:21.021306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49074 (local address 127.8.249.190:40117)
0926 20:52:21.021480 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:21.021485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:21.021989 (+ 504us) server_negotiation.cc:407] Connection header received
0926 20:52:21.022141 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:21.022145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:21.022207 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:21.022305 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:21.023571 (+ 1266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49074: BlockingRecv error: recv got EOF from 127.0.0.1:49074 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:52:21.026850 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 11710
W20260926 20:52:21.029356 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 11710: Not authorized: Client connection negotiation failed: client connection to 127.8.249.129:38749: server requires authentication, but client does not have Kerberos credentials available
I20260926 20:52:21.029543 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 11710
I20260926 20:52:21.044983 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 11844
W20260926 20:52:21.047811 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 11844: Not authorized: Client connection negotiation failed: client connection to 127.8.249.130:42525: server requires authentication, but client does not have Kerberos credentials available
I20260926 20:52:21.047986 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 11844
I20260926 20:52:21.063026 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 11977
W20260926 20:52:21.066115 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 11977: Not authorized: Client connection negotiation failed: client connection to 127.8.249.131:40127: server requires authentication, but client does not have Kerberos credentials available
I20260926 20:52:21.066257 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 11977
I20260926 20:52:21.081437 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 11617
I20260926 20:52:21.082831 11679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:21.184371 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 11617
2026-09-26T20:52:21Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3660 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:52:21 dist-test-slave-2dg3 krb5kdc[12122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:52:21 dist-test-slave-2dg3 krb5kdc[12122](info): set up 2 sockets
Sep 26 20:52:21 dist-test-slave-2dg3 krb5kdc[12122](info): commencing operation
krb5kdc: starting...
W20260926 20:52:22.271163 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:52:22 dist-test-slave-2dg3 krb5kdc[12122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:52:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:52:22.469849 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:44265
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:44265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:22.792505 12135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:22.792781 12135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:22.792886 12135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:22.800033 12135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:22.800148 12135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:22.800194 12135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:22.800244 12135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:22.800292 12135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:22.810009 12135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:44265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:44265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:22.811288 12135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:22.812530 12135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:22.824575 12140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:22.824643 12143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:22.824608 12141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:22.825431 12135 server_base.cc:1034] running on GCE node
I20260926 20:52:22.826282 12135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:22.827899 12135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:22.829123 12135 hybrid_clock.cc:648] HybridClock initialized: now 1790455942829085 us; error 38 us; skew 500 ppm
I20260926 20:52:22.829465 12135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:22 dist-test-slave-2dg3 krb5kdc[12122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455942, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:22.838563 12135 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:52:22.840212 12135 webserver.cc:460] Webserver started at http://127.8.249.190:43365/ using document root <none> and password file <none>
I20260926 20:52:22.840948 12135 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:22.841105 12135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:22.841445 12135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:22.844569 12135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "deeba5eecf664afba49b32c9aee59628"
format_stamp: "Formatted at 2026-09-26 20:52:22 on dist-test-slave-2dg3"
server_key: "fd93406679a38b77561dcd791c45b35f"
server_key_iv: "ae579a7e5d99245ce26fd3970b2308cd"
server_key_version: "encryptionkey@0"
I20260926 20:52:22.845383 12135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "deeba5eecf664afba49b32c9aee59628"
format_stamp: "Formatted at 2026-09-26 20:52:22 on dist-test-slave-2dg3"
server_key: "fd93406679a38b77561dcd791c45b35f"
server_key_iv: "ae579a7e5d99245ce26fd3970b2308cd"
server_key_version: "encryptionkey@0"
I20260926 20:52:22.850883 12135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 20:52:22.854197 12150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:22.855567 12135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 20:52:22.855741 12135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "deeba5eecf664afba49b32c9aee59628"
format_stamp: "Formatted at 2026-09-26 20:52:22 on dist-test-slave-2dg3"
server_key: "fd93406679a38b77561dcd791c45b35f"
server_key_iv: "ae579a7e5d99245ce26fd3970b2308cd"
server_key_version: "encryptionkey@0"
I20260926 20:52:22.855945 12135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:22.873245 12135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:22.877609 12135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:22.877894 12135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:22.896448 12135 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:44265
I20260926 20:52:22.896445 12201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:44265 every 8 connection(s)
I20260926 20:52:22.897859 12135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:22.899139 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 12135
I20260926 20:52:22.899338 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:22.899942 9190 external_mini_cluster.cc:1428] Setting key d7b96a4c5389a15d7c37e753366f9975
I20260926 20:52:22.902199 12202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 20:52:22 dist-test-slave-2dg3 krb5kdc[12122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:22.915475 12202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628: Bootstrap starting.
I20260926 20:52:22.918648 12202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:22.919709 12202 log.cc:824] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:22.922434 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:22.901679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.8.249.190:44265)
0926 20:52:22.902448 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:22.902463 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:22.902502 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:52:22.904105 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:22.904126 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:22.904506 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:22.904837 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:22.906629 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:22.908040 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:22.909787 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:22.910467 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:22.910595 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:22.915281 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:22.915289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:22.915306 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:22.915345 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:22.918386 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:22.919185 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:22.919191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:22.919197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:22.919352 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:22.919720 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:22.919726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:22.919729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:22.920407 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:22.920737 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:22.921137 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 20:52:22.921535 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":357,"threads_started":1}
I20260926 20:52:22.922712 12202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628: No bootstrap required, opened a new log
I20260926 20:52:22.926151 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:52:22.930408 12202 raft_consensus.cc:348] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deeba5eecf664afba49b32c9aee59628" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44265 } }
I20260926 20:52:22.930882 12202 raft_consensus.cc:374] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:22.931002 12202 raft_consensus.cc:729] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deeba5eecf664afba49b32c9aee59628, State: Initialized, Role: FOLLOWER
I20260926 20:52:22.931602 12202 consensus_queue.cc:260] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deeba5eecf664afba49b32c9aee59628" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44265 } }
I20260926 20:52:22.931861 12202 raft_consensus.cc:388] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:22.932011 12202 raft_consensus.cc:482] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:22.932183 12202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:22.934410 12202 raft_consensus.cc:504] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deeba5eecf664afba49b32c9aee59628" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44265 } }
I20260926 20:52:22.935014 12202 leader_election.cc:302] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deeba5eecf664afba49b32c9aee59628; no voters:
I20260926 20:52:22.935689 12202 leader_election.cc:288] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:22.935829 12208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:22.936978 12208 raft_consensus.cc:686] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [term 1 LEADER]: Becoming Leader. State: Replica: deeba5eecf664afba49b32c9aee59628, State: Running, Role: LEADER
I20260926 20:52:22.937613 12208 consensus_queue.cc:237] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deeba5eecf664afba49b32c9aee59628" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44265 } }
I20260926 20:52:22.938141 12202 sys_catalog.cc:564] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:22.940744 12210 sys_catalog.cc:455] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [sys.catalog]: SysCatalogTable state changed. Reason: New leader deeba5eecf664afba49b32c9aee59628. Latest consensus state: current_term: 1 leader_uuid: "deeba5eecf664afba49b32c9aee59628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deeba5eecf664afba49b32c9aee59628" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44265 } } }
I20260926 20:52:22.941211 12210 sys_catalog.cc:458] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:22.941571 12209 sys_catalog.cc:455] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deeba5eecf664afba49b32c9aee59628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deeba5eecf664afba49b32c9aee59628" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44265 } } }
I20260926 20:52:22.942226 12209 sys_catalog.cc:458] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:22.948552 12217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:22.954185 12217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:22.964370 12217 catalog_manager.cc:1347] Generated new cluster ID: 4bb1c27d80104b1b964a7aabb8679f1a
I20260926 20:52:22.964502 12217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:22.979676 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:22.965087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.8.249.190:44265)
0926 20:52:22.970992 (+ 5905us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:22.970999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:22.971580 (+ 581us) server_negotiation.cc:407] Connection header received
0926 20:52:22.971761 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:22.971766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:22.971838 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:22.971960 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:22.973134 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:22.974210 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:22.976068 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:22.976680 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:22.976808 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:22.977979 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:22.977985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:22.977990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:22.978885 (+ 895us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:22.978944 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:22.978975 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:22.979068 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:22.979226 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 20:52:22.979313 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5770}
I20260926 20:52:22.999416 12217 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:23.001458 12217 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:23.026321 12217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P deeba5eecf664afba49b32c9aee59628: Generated new TSK 0
I20260926 20:52:23.027285 12217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:52:23.044502 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 12135
I20260926 20:52:23.046003 12197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:23.138576 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 12135
2026-09-26T20:52:23Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1948 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:52:23 dist-test-slave-2dg3 krb5kdc[12236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:52:23 dist-test-slave-2dg3 krb5kdc[12236](info): set up 2 sockets
Sep 26 20:52:23 dist-test-slave-2dg3 krb5kdc[12236](info): commencing operation
krb5kdc: starting...
W20260926 20:52:24.211233 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.010s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:52:24 dist-test-slave-2dg3 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:52:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:52:24.410053 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36739
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:36739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:24.735735 12249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:24.736097 12249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:24.736253 12249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:24.743533 12249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:24.743680 12249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:24.743784 12249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:24.743889 12249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:24.743974 12249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:24.753834 12249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:36739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:24.755208 12249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:24.756528 12249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:24.768155 12255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:24.768107 12257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:24.768591 12254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:24.768863 12249 server_base.cc:1034] running on GCE node
I20260926 20:52:24.769622 12249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:24.771039 12249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:24.772295 12249 hybrid_clock.cc:648] HybridClock initialized: now 1790455944772248 us; error 44 us; skew 500 ppm
I20260926 20:52:24.772717 12249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:24 dist-test-slave-2dg3 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455944, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:24.775942 12249 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:52:24.777395 12249 webserver.cc:460] Webserver started at http://127.8.249.190:42661/ using document root <none> and password file <none>
I20260926 20:52:24.778052 12249 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:24.778187 12249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:24.778469 12249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:24.781430 12249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "665de10031204b2fb4465c1e7e74f968"
format_stamp: "Formatted at 2026-09-26 20:52:24 on dist-test-slave-2dg3"
server_key: "b575664091011ee47350b04b41a8597d"
server_key_iv: "664f91b2269c9eeac8b1f4b47ab8b75b"
server_key_version: "encryptionkey@0"
I20260926 20:52:24.782162 12249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "665de10031204b2fb4465c1e7e74f968"
format_stamp: "Formatted at 2026-09-26 20:52:24 on dist-test-slave-2dg3"
server_key: "b575664091011ee47350b04b41a8597d"
server_key_iv: "664f91b2269c9eeac8b1f4b47ab8b75b"
server_key_version: "encryptionkey@0"
I20260926 20:52:24.787446 12249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 20:52:24.790973 12264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:24.792277 12249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:52:24.792471 12249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "665de10031204b2fb4465c1e7e74f968"
format_stamp: "Formatted at 2026-09-26 20:52:24 on dist-test-slave-2dg3"
server_key: "b575664091011ee47350b04b41a8597d"
server_key_iv: "664f91b2269c9eeac8b1f4b47ab8b75b"
server_key_version: "encryptionkey@0"
I20260926 20:52:24.792686 12249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:24.805663 12249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:24.810452 12249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:24.810783 12249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:24.830395 12249 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:36739
I20260926 20:52:24.830412 12315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:36739 every 8 connection(s)
I20260926 20:52:24.831900 12249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:24.835862 12316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:24.839705 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 12249
I20260926 20:52:24.839939 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:24.840718 9190 external_mini_cluster.cc:1428] Setting key 9f5f4c6abb2b34ce597a9a616b827357
I20260926 20:52:24.849416 12316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968: Bootstrap starting.
I20260926 20:52:24.853328 12316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:24.854684 12316 log.cc:824] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:52:24 dist-test-slave-2dg3 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:24.857604 12316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968: No bootstrap required, opened a new log
I20260926 20:52:24.863353 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:24.842827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.8.249.190:36739)
0926 20:52:24.843624 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:24.843637 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:24.843678 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:52:24.845206 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:24.845227 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:24.845642 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:24.845969 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:24.847725 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:24.848964 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:24.850660 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:24.851532 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:24.851675 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:24.856178 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:24.856185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:24.856200 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:24.856235 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:24.859265 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:24.859993 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:24.860000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:24.860005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:24.860175 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:24.860555 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:24.860561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:24.860565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:24.861115 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:24.861434 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:24.861873 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 20:52:24.862281 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":373,"threads_started":1}
I20260926 20:52:24.865545 12316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665de10031204b2fb4465c1e7e74f968" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36739 } }
I20260926 20:52:24.865947 12316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:24.866101 12316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665de10031204b2fb4465c1e7e74f968, State: Initialized, Role: FOLLOWER
I20260926 20:52:24.866688 12316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665de10031204b2fb4465c1e7e74f968" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36739 } }
I20260926 20:52:24.867055 12316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:24.867213 12316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:24.867396 12316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:24.869488 12316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665de10031204b2fb4465c1e7e74f968" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36739 } }
I20260926 20:52:24.869956 12316 leader_election.cc:302] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 665de10031204b2fb4465c1e7e74f968; no voters:
I20260926 20:52:24.870615 12316 leader_election.cc:288] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:24.870705 12321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:24.871374 12321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [term 1 LEADER]: Becoming Leader. State: Replica: 665de10031204b2fb4465c1e7e74f968, State: Running, Role: LEADER
I20260926 20:52:24.871927 12321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665de10031204b2fb4465c1e7e74f968" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36739 } }
I20260926 20:52:24.872699 12316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:24.877069 12322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "665de10031204b2fb4465c1e7e74f968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665de10031204b2fb4465c1e7e74f968" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36739 } } }
I20260926 20:52:24.877702 12322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:24.877048 12323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 665de10031204b2fb4465c1e7e74f968. Latest consensus state: current_term: 1 leader_uuid: "665de10031204b2fb4465c1e7e74f968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665de10031204b2fb4465c1e7e74f968" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36739 } } }
I20260926 20:52:24.878187 12323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:24.880216 12333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:24.885505 12333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:24.893500 12333 catalog_manager.cc:1347] Generated new cluster ID: b1e7806a2a954bdfad75366c4376b8c1
I20260926 20:52:24.893646 12333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:24.909276 12333 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:24.910998 12333 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:24.919476 12333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 665de10031204b2fb4465c1e7e74f968: Generated new TSK 0
I20260926 20:52:24.920306 12333 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:52:24.968613 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36739
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:25.300632 12342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:25.300945 12342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:25.301108 12342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:25.308656 12342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:25.308779 12342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:25.308995 12342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:52:25.318651 12342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:25.319993 12342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:25.321295 12342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:25.334857 12348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:25.334893 12347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:25.335127 12350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:25.336205 12342 server_base.cc:1034] running on GCE node
I20260926 20:52:25.336819 12342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:25.337746 12342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:25.338972 12342 hybrid_clock.cc:648] HybridClock initialized: now 1790455945338913 us; error 70 us; skew 500 ppm
I20260926 20:52:25.339288 12342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:25 dist-test-slave-2dg3 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455945, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:25.342303 12342 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:52:25.343835 12342 webserver.cc:460] Webserver started at http://127.8.249.129:38447/ using document root <none> and password file <none>
I20260926 20:52:25.344507 12342 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:25.344650 12342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:25.344954 12342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:25.347828 12342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "ddd535061f38451fa33ab3140be4f36d"
format_stamp: "Formatted at 2026-09-26 20:52:25 on dist-test-slave-2dg3"
server_key: "8e113e6a95368dee091c2c9d702c6d81"
server_key_iv: "fec785ef90c6663f86962eb7c345fb26"
server_key_version: "encryptionkey@0"
I20260926 20:52:25.348570 12342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "ddd535061f38451fa33ab3140be4f36d"
format_stamp: "Formatted at 2026-09-26 20:52:25 on dist-test-slave-2dg3"
server_key: "8e113e6a95368dee091c2c9d702c6d81"
server_key_iv: "fec785ef90c6663f86962eb7c345fb26"
server_key_version: "encryptionkey@0"
I20260926 20:52:25.353633 12342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:52:25.357111 12357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:25.358453 12342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:52:25.358630 12342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "ddd535061f38451fa33ab3140be4f36d"
format_stamp: "Formatted at 2026-09-26 20:52:25 on dist-test-slave-2dg3"
server_key: "8e113e6a95368dee091c2c9d702c6d81"
server_key_iv: "fec785ef90c6663f86962eb7c345fb26"
server_key_version: "encryptionkey@0"
I20260926 20:52:25.358871 12342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:25.374621 12342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:25.378492 12342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:25.378784 12342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:25.379851 12342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:25.381132 12342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:25.381234 12342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:25.381377 12342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:25.381454 12342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:25.413761 12342 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:44663
I20260926 20:52:25.413800 12469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:44663 every 8 connection(s)
I20260926 20:52:25.415158 12342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:52:25.419051 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 12342
I20260926 20:52:25.419267 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:52:25.420038 9190 external_mini_cluster.cc:1428] Setting key a43b1440bf1ca7c4233606b75a0647ab
Sep 26 20:52:25 dist-test-slave-2dg3 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455945, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:25.435196 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:25.417660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:52605 (local address 127.8.249.190:36739)
0926 20:52:25.418130 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:25.418136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:25.419323 (+ 1187us) server_negotiation.cc:407] Connection header received
0926 20:52:25.421438 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:25.421445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:25.421566 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:25.421682 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:25.423324 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:25.424015 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:25.425586 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:25.425973 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:25.426103 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:25.429669 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:25.429674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:25.429677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:25.429707 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:25.432254 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:25.432975 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:25.432981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:25.432983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:25.433092 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:25.433507 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:25.433512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:25.433515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:25.433769 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:25.433927 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:25.434597 (+ 670us) server_negotiation.cc:299] Negotiation successful
0926 20:52:25.434839 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":229,"threads_started":1}
I20260926 20:52:25.436159 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:25.418322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36739 (local address 127.8.249.129:52605)
0926 20:52:25.419165 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:25.419218 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:25.421112 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:25.422009 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:25.422019 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:25.422527 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:25.423154 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:25.423172 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:25.424191 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:25.424195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:25.425414 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:25.425425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:25.426201 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:25.426207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:25.426441 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:25.427146 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:25.427168 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:25.429458 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:25.432406 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:25.432415 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:25.432432 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:25.432845 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:25.433201 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:25.433206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:25.433210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:25.433393 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:25.433893 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:25.433902 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:25.434239 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 20:52:25.434491 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 20:52:25.435055 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":336,"threads_started":1}
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
I20260926 20:52:25.437794 12470 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36739
I20260926 20:52:25.438107 12470 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:25.438932 12470 heartbeater.cc:511] Master 127.8.249.190:36739 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
I20260926 20:52:25.441063 12281 ts_manager.cc:194] Registered new tserver with Master: ddd535061f38451fa33ab3140be4f36d (127.8.249.129:44663)
I20260926 20:52:25.442488 12281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:52605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:52:25.464650 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36739
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:25.784045 12476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:25.784313 12476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:25.784487 12476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:25.791673 12476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:25.791810 12476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:25.792016 12476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:52:25.801865 12476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:25.803262 12476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:25.804564 12476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:25.817618 12481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:25.817654 12484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:25.817556 12482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:25.818527 12476 server_base.cc:1034] running on GCE node
I20260926 20:52:25.819293 12476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:25.820232 12476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:25.821427 12476 hybrid_clock.cc:648] HybridClock initialized: now 1790455945821375 us; error 35 us; skew 500 ppm
I20260926 20:52:25.821741 12476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:25 dist-test-slave-2dg3 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455945, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:25.824855 12476 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:52:25.826267 12476 webserver.cc:460] Webserver started at http://127.8.249.130:41623/ using document root <none> and password file <none>
I20260926 20:52:25.827247 12476 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:25.827425 12476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:25.827754 12476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:25.830711 12476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "e97223bfe1974756be509c50a294d921"
format_stamp: "Formatted at 2026-09-26 20:52:25 on dist-test-slave-2dg3"
server_key: "68e38a2ffcdd37c865b871dd5330e951"
server_key_iv: "6a90339ad172da610ce59c095b8b24bb"
server_key_version: "encryptionkey@0"
I20260926 20:52:25.831565 12476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "e97223bfe1974756be509c50a294d921"
format_stamp: "Formatted at 2026-09-26 20:52:25 on dist-test-slave-2dg3"
server_key: "68e38a2ffcdd37c865b871dd5330e951"
server_key_iv: "6a90339ad172da610ce59c095b8b24bb"
server_key_version: "encryptionkey@0"
I20260926 20:52:25.836874 12476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:52:25.840755 12491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:25.842190 12476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:52:25.842391 12476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "e97223bfe1974756be509c50a294d921"
format_stamp: "Formatted at 2026-09-26 20:52:25 on dist-test-slave-2dg3"
server_key: "68e38a2ffcdd37c865b871dd5330e951"
server_key_iv: "6a90339ad172da610ce59c095b8b24bb"
server_key_version: "encryptionkey@0"
I20260926 20:52:25.842638 12476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:25.864578 12476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:25.868671 12476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:25.868983 12476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:25.869930 12476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:25.871289 12476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:25.871389 12476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:25.871529 12476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:25.871598 12476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:25.904659 12476 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:37251
I20260926 20:52:25.904711 12603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:37251 every 8 connection(s)
I20260926 20:52:25.905961 12476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:52:25.914017 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 12476
I20260926 20:52:25.914216 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:52:25.914777 9190 external_mini_cluster.cc:1428] Setting key 42c9a005d6f71de24f925bf7791ac37b
Sep 26 20:52:25 dist-test-slave-2dg3 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455945, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:25.926988 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:25.908512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:39495 (local address 127.8.249.190:36739)
0926 20:52:25.908730 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:25.908736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:25.909833 (+ 1097us) server_negotiation.cc:407] Connection header received
0926 20:52:25.911574 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:25.911578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:25.911644 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:25.911730 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:25.913128 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:25.913932 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:25.916120 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:25.916791 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:25.916932 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:25.921259 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:25.921264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:25.921267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:25.921299 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:25.923834 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:25.924626 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:25.924630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:25.924633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:25.924755 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:25.925199 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:25.925207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:25.925211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:25.925503 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:25.925658 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:25.926364 (+ 706us) server_negotiation.cc:299] Negotiation successful
0926 20:52:25.926597 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:52:25.928009 12606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:25.908981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36739 (local address 127.8.249.130:39495)
0926 20:52:25.909657 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:25.909695 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:25.911326 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:25.911932 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:25.911940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:25.912437 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:25.912978 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:25.912994 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:25.914098 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:25.914105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:25.915977 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:25.915989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:25.917040 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:25.917048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:25.917365 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:25.918118 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:25.918155 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:25.921007 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:25.924003 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:25.924013 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:25.924033 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:25.924484 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:25.924868 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:25.924873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:25.924877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:25.925077 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:25.925634 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:25.925643 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:25.926001 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 20:52:25.926279 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 20:52:25.926880 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":262,"threads_started":1}
I20260926 20:52:25.929630 12604 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36739
I20260926 20:52:25.930004 12604 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:25.930728 12604 heartbeater.cc:511] Master 127.8.249.190:36739 requested a full tablet report, sending...
I20260926 20:52:25.932268 12281 ts_manager.cc:194] Registered new tserver with Master: e97223bfe1974756be509c50a294d921 (127.8.249.130:37251)
I20260926 20:52:25.933074 12281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:39495
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:52:25.962945 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36739
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:26.295135 12609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:26.295435 12609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:26.295529 12609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:26.302719 12609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:26.302846 12609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:26.302958 12609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:52:26.312709 12609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:26.313941 12609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:26.315310 12609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:26.328012 12614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:26.328819 12615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:26.328969 12617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:26.329645 12609 server_base.cc:1034] running on GCE node
I20260926 20:52:26.330215 12609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:26.331234 12609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:26.332432 12609 hybrid_clock.cc:648] HybridClock initialized: now 1790455946332383 us; error 37 us; skew 500 ppm
I20260926 20:52:26.332758 12609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:26 dist-test-slave-2dg3 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455946, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:26.335832 12609 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:52:26.337296 12609 webserver.cc:460] Webserver started at http://127.8.249.131:42481/ using document root <none> and password file <none>
I20260926 20:52:26.337988 12609 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:26.338136 12609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:26.338450 12609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:26.341382 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "bb64b7ed4d6c495b93fe917af66a468d"
format_stamp: "Formatted at 2026-09-26 20:52:26 on dist-test-slave-2dg3"
server_key: "cd304cd156439dde4db67ff6e8226015"
server_key_iv: "b4c5dbe62522a0a0d05d970823ae4d06"
server_key_version: "encryptionkey@0"
I20260926 20:52:26.342180 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "bb64b7ed4d6c495b93fe917af66a468d"
format_stamp: "Formatted at 2026-09-26 20:52:26 on dist-test-slave-2dg3"
server_key: "cd304cd156439dde4db67ff6e8226015"
server_key_iv: "b4c5dbe62522a0a0d05d970823ae4d06"
server_key_version: "encryptionkey@0"
I20260926 20:52:26.347476 12609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 20:52:26.351063 12624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:26.352514 12609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:52:26.352710 12609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "bb64b7ed4d6c495b93fe917af66a468d"
format_stamp: "Formatted at 2026-09-26 20:52:26 on dist-test-slave-2dg3"
server_key: "cd304cd156439dde4db67ff6e8226015"
server_key_iv: "b4c5dbe62522a0a0d05d970823ae4d06"
server_key_version: "encryptionkey@0"
I20260926 20:52:26.352924 12609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:26.369076 12609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:26.373067 12609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:26.373354 12609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:26.374315 12609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:26.375761 12609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:26.375893 12609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:26.376060 12609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:26.376111 12609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:26.409034 12609 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:38789
I20260926 20:52:26.409070 12736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:38789 every 8 connection(s)
I20260926 20:52:26.410287 12609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:52:26.412591 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 12609
I20260926 20:52:26.412770 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:52:26.413326 9190 external_mini_cluster.cc:1428] Setting key e71a66fb7c69b7f4679c55dcc2084a3f
Sep 26 20:52:26 dist-test-slave-2dg3 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455946, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:26.430217 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:26.412810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:43041 (local address 127.8.249.190:36739)
0926 20:52:26.413026 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:26.413033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:26.414400 (+ 1367us) server_negotiation.cc:407] Connection header received
0926 20:52:26.416756 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:26.416761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:26.416847 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:26.416966 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:26.418419 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:26.419223 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:26.420720 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:26.421130 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:26.421254 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:26.424607 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:26.424612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:26.424616 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:26.424648 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:26.427176 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:26.428052 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:26.428056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:26.428059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:26.428209 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:26.428751 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:26.428756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:26.428759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:26.429097 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:26.429204 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:26.429826 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 20:52:26.429981 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:52:26.431265 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:26.413392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36739 (local address 127.8.249.131:43041)
0926 20:52:26.414252 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:26.414304 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:26.416510 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:26.417190 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:26.417200 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:26.417680 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:26.418247 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:26.418263 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:26.419385 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:26.419389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:26.420577 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:26.420588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:26.421351 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:26.421357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:26.421591 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:26.422250 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:26.422272 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:26.424415 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:26.427310 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:26.427319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:26.427337 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:26.427943 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:26.428329 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:26.428337 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:26.428340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:26.428635 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:26.429212 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:26.429218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:26.429525 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 20:52:26.429752 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 20:52:26.430136 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":329,"threads_started":1}
I20260926 20:52:26.432571 12737 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36739
I20260926 20:52:26.432860 12737 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:26.433580 12737 heartbeater.cc:511] Master 127.8.249.190:36739 requested a full tablet report, sending...
I20260926 20:52:26.435038 12281 ts_manager.cc:194] Registered new tserver with Master: bb64b7ed4d6c495b93fe917af66a468d (127.8.249.131:38789)
I20260926 20:52:26.435819 12281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:43041
I20260926 20:52:26.441120 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:52:26.445667 12470 heartbeater.cc:503] Master 127.8.249.190:36739 was elected leader, sending a full tablet report...
Sep 26 20:52:26 dist-test-slave-2dg3 krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455946, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 20:52:26 dist-test-slave-2dg3 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455946, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:26.489889 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:26.476506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.8.249.190:36739)
0926 20:52:26.476688 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:26.476693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:26.477152 (+ 459us) server_negotiation.cc:407] Connection header received
0926 20:52:26.477309 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:26.477313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:26.477377 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:26.477473 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:26.478917 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:26.479692 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:26.480860 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:26.481250 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:26.481340 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:26.485076 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:26.485081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:26.485089 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:26.485139 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:26.487744 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:26.488312 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:26.488317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:26.488319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:26.488456 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:26.488776 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:26.488780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:26.488782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:26.489095 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:26.489285 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:26.489502 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 20:52:26.489660 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 20:52:26.490454 12281 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49808
I20260926 20:52:26.492254 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 12342
Sep 26 20:52:26 dist-test-slave-2dg3 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455946, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:52:26.507440 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:26.493198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.8.249.129:44663)
0926 20:52:26.493796 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:26.493804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:26.493836 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:52:26.493887 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:26.493894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:26.494082 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:26.494260 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:26.495461 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:26.496532 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:26.497763 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:26.498326 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:26.498478 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:26.502159 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:26.502169 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:26.502184 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:26.502218 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:26.504994 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:26.505552 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:26.505558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:26.505560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:26.505672 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:26.505970 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:26.505973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:26.505974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:26.506272 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:26.506487 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:26.506841 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 20:52:26.507035 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":247,"threads_started":1}
W20260926 20:52:26.508597 12465 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45118
W20260926 20:52:26.509673 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 12342: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:52:26.509894 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 12342
I20260926 20:52:26.526466 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 12476
Sep 26 20:52:26 dist-test-slave-2dg3 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455946, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:52:26.541625 12749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:26.527354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.8.249.130:37251)
0926 20:52:26.527959 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:26.527966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:26.527992 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:52:26.528052 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:26.528058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:26.528274 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:26.528413 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:26.529562 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:26.530576 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:26.531912 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:26.532469 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:26.532562 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:26.536412 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:26.536423 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:26.536437 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:26.536473 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:26.539290 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:26.539853 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:26.539859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:26.539862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:26.539972 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:26.540268 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:26.540271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:26.540273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:26.540589 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:26.540800 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:26.541033 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 20:52:26.541220 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":312,"threads_started":1}
W20260926 20:52:26.542879 12599 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35174
W20260926 20:52:26.543964 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 12476: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:52:26.544173 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 12476
I20260926 20:52:26.560195 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 12609
Sep 26 20:52:26 dist-test-slave-2dg3 krb5kdc[12236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455946, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:52:26.576912 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:26.561245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.8.249.131:38789)
0926 20:52:26.561814 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:26.561821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:26.561848 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:52:26.561907 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:26.561914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:26.562124 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:26.562303 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:26.563668 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:26.564682 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:26.566005 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:26.566564 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:26.566702 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:26.570919 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:26.570931 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:26.570947 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:26.570980 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:26.573998 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:26.574740 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:26.574748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:26.574750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:26.574926 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:26.575290 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:26.575296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:26.575300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:26.575659 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:26.575955 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:26.576283 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 20:52:26.576564 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":293,"threads_started":1}
W20260926 20:52:26.578274 12732 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34020
W20260926 20:52:26.579641 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 12609: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:52:26.579823 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 12609
I20260926 20:52:26.595644 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 12249
I20260926 20:52:26.600783 12311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:26.706090 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 12249
2026-09-26T20:52:26Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3573 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:52:26 dist-test-slave-2dg3 krb5kdc[12754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:52:26 dist-test-slave-2dg3 krb5kdc[12754](info): set up 2 sockets
Sep 26 20:52:26 dist-test-slave-2dg3 krb5kdc[12754](info): commencing operation
krb5kdc: starting...
W20260926 20:52:27.775261 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:52:27 dist-test-slave-2dg3 krb5kdc[12754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:52:27.975145 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:43759
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:43759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:28.289112 12767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:28.289449 12767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:28.289616 12767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:28.296783 12767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:28.296955 12767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:28.297065 12767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:28.297168 12767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:28.297250 12767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:28.307104 12767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:43759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:43759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:28.308534 12767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:28.309903 12767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:28.321285 12772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:28.321388 12775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:28.321357 12773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:28.322448 12767 server_base.cc:1034] running on GCE node
I20260926 20:52:28.323257 12767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:28.324641 12767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:28.325871 12767 hybrid_clock.cc:648] HybridClock initialized: now 1790455948325835 us; error 48 us; skew 500 ppm
I20260926 20:52:28.326211 12767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:28 dist-test-slave-2dg3 krb5kdc[12754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455948, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:28.335009 12767 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:52:28.336447 12767 webserver.cc:460] Webserver started at http://127.8.249.190:38093/ using document root <none> and password file <none>
I20260926 20:52:28.337175 12767 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:28.337317 12767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:28.337623 12767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:28.340523 12767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "7e0ac3b54e4e4363981384282b8dc6e3"
format_stamp: "Formatted at 2026-09-26 20:52:28 on dist-test-slave-2dg3"
server_key: "ef55b7fad08fbeaa385f99c46e2890f7"
server_key_iv: "0da4dc4fdb59b077349c812e6c22125f"
server_key_version: "encryptionkey@0"
I20260926 20:52:28.341292 12767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "7e0ac3b54e4e4363981384282b8dc6e3"
format_stamp: "Formatted at 2026-09-26 20:52:28 on dist-test-slave-2dg3"
server_key: "ef55b7fad08fbeaa385f99c46e2890f7"
server_key_iv: "0da4dc4fdb59b077349c812e6c22125f"
server_key_version: "encryptionkey@0"
I20260926 20:52:28.346231 12767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 20:52:28.349606 12782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:28.350898 12767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:28.351083 12767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "7e0ac3b54e4e4363981384282b8dc6e3"
format_stamp: "Formatted at 2026-09-26 20:52:28 on dist-test-slave-2dg3"
server_key: "ef55b7fad08fbeaa385f99c46e2890f7"
server_key_iv: "0da4dc4fdb59b077349c812e6c22125f"
server_key_version: "encryptionkey@0"
I20260926 20:52:28.351279 12767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:28.364669 12767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:28.368456 12767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:28.368738 12767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:28.386554 12767 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:43759
I20260926 20:52:28.386572 12833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:43759 every 8 connection(s)
I20260926 20:52:28.388015 12767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:28.391633 12834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:28.394429 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 12767
I20260926 20:52:28.394658 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:28.395438 9190 external_mini_cluster.cc:1428] Setting key c57f9dd0faa594801275b3ee4402badd
I20260926 20:52:28.404060 12834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3: Bootstrap starting.
I20260926 20:52:28.407927 12834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:28.409222 12834 log.cc:824] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:52:28 dist-test-slave-2dg3 krb5kdc[12754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:28.412218 12834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3: No bootstrap required, opened a new log
I20260926 20:52:28.418416 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:28.397556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.8.249.190:43759)
0926 20:52:28.398332 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:28.398344 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:28.398386 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:52:28.400063 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:28.400084 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:28.400472 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:28.400804 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:28.402489 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:28.403925 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:28.405617 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:28.406328 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:28.406456 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:28.411372 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:28.411379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:28.411394 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:28.411427 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:28.414499 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:28.415232 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:28.415245 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:28.415251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:28.415408 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:28.415774 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:28.415778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:28.415781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:28.416293 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:28.416635 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:28.417047 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 20:52:28.417434 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":361,"threads_started":1}
I20260926 20:52:28.420042 12834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0ac3b54e4e4363981384282b8dc6e3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43759 } }
I20260926 20:52:28.420389 12834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:28.420512 12834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0ac3b54e4e4363981384282b8dc6e3, State: Initialized, Role: FOLLOWER
I20260926 20:52:28.421051 12834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0ac3b54e4e4363981384282b8dc6e3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43759 } }
I20260926 20:52:28.421301 12834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:28.421428 12834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:28.421545 12834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:28.423653 12834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0ac3b54e4e4363981384282b8dc6e3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43759 } }
I20260926 20:52:28.424191 12834 leader_election.cc:302] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e0ac3b54e4e4363981384282b8dc6e3; no voters:
I20260926 20:52:28.424860 12834 leader_election.cc:288] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:28.425019 12839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:28.425750 12839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [term 1 LEADER]: Becoming Leader. State: Replica: 7e0ac3b54e4e4363981384282b8dc6e3, State: Running, Role: LEADER
I20260926 20:52:28.426267 12839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0ac3b54e4e4363981384282b8dc6e3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43759 } }
I20260926 20:52:28.426738 12834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:28.429643 12841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e0ac3b54e4e4363981384282b8dc6e3. Latest consensus state: current_term: 1 leader_uuid: "7e0ac3b54e4e4363981384282b8dc6e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0ac3b54e4e4363981384282b8dc6e3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43759 } } }
I20260926 20:52:28.429611 12840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e0ac3b54e4e4363981384282b8dc6e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0ac3b54e4e4363981384282b8dc6e3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43759 } } }
I20260926 20:52:28.430115 12841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:28.430171 12840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:28.433769 12846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:28.438290 12846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:28.446925 12846 catalog_manager.cc:1347] Generated new cluster ID: b2c792affcc547acab99114eed7bff24
I20260926 20:52:28.447078 12846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:28.467536 12846 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:28.468886 12846 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:28.475574 12846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e0ac3b54e4e4363981384282b8dc6e3: Generated new TSK 0
I20260926 20:52:28.476471 12846 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:52:28.531183 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:43759
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:28.870239 12860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:28.870581 12860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:28.870749 12860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:28.878158 12860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:28.878315 12860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:28.878495 12860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:52:28.888624 12860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:43759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:28.890058 12860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:28.891435 12860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:28.905006 12865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:28.905092 12866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:28.905097 12868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:28.905915 12860 server_base.cc:1034] running on GCE node
I20260926 20:52:28.906595 12860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:28.907786 12860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:28.909039 12860 hybrid_clock.cc:648] HybridClock initialized: now 1790455948908966 us; error 66 us; skew 500 ppm
I20260926 20:52:28.909416 12860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:28 dist-test-slave-2dg3 krb5kdc[12754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455948, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:28.912604 12860 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:52:28.914101 12860 webserver.cc:460] Webserver started at http://127.8.249.129:41473/ using document root <none> and password file <none>
I20260926 20:52:28.914863 12860 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:28.915012 12860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:28.915318 12860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:28.918215 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "e6183f252915475dbdcc5201f8e71cd2"
format_stamp: "Formatted at 2026-09-26 20:52:28 on dist-test-slave-2dg3"
server_key: "a1147a7dda90a2144f8abd44b04db81a"
server_key_iv: "0e1d7445d17daedd05a2eec8aff1eb9b"
server_key_version: "encryptionkey@0"
I20260926 20:52:28.919077 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "e6183f252915475dbdcc5201f8e71cd2"
format_stamp: "Formatted at 2026-09-26 20:52:28 on dist-test-slave-2dg3"
server_key: "a1147a7dda90a2144f8abd44b04db81a"
server_key_iv: "0e1d7445d17daedd05a2eec8aff1eb9b"
server_key_version: "encryptionkey@0"
I20260926 20:52:28.924113 12860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 20:52:28.927608 12875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:28.929045 12860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:52:28.929234 12860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "e6183f252915475dbdcc5201f8e71cd2"
format_stamp: "Formatted at 2026-09-26 20:52:28 on dist-test-slave-2dg3"
server_key: "a1147a7dda90a2144f8abd44b04db81a"
server_key_iv: "0e1d7445d17daedd05a2eec8aff1eb9b"
server_key_version: "encryptionkey@0"
I20260926 20:52:28.929448 12860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:28.947191 12860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:28.951418 12860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:28.951701 12860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:28.952642 12860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:28.954020 12860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:28.954121 12860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:28.954267 12860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:28.954351 12860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:28.987298 12860 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:35679
I20260926 20:52:28.987362 12987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:35679 every 8 connection(s)
I20260926 20:52:28.988600 12860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:52:28.990358 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 12860
I20260926 20:52:28.990595 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:52:28.991369 9190 external_mini_cluster.cc:1428] Setting key 8b3e5057f0ba883e65a0976e9a679230
Sep 26 20:52:28 dist-test-slave-2dg3 krb5kdc[12754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455948, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:29.011895 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:28.991225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:33225 (local address 127.8.249.190:43759)
0926 20:52:28.991682 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:28.991688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:28.992786 (+ 1098us) server_negotiation.cc:407] Connection header received
0926 20:52:28.995410 (+ 2624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:28.995413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:28.995479 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:28.995570 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:28.997609 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:28.998306 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:29.000358 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:29.001108 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:29.001251 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:29.004723 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:29.004732 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:29.004740 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:29.004796 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:29.008388 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:29.009215 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:29.009223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:29.009230 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:29.009397 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:29.009838 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:29.009843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:29.009848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:29.010227 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:29.010390 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:29.011163 (+ 773us) server_negotiation.cc:299] Negotiation successful
0926 20:52:29.011420 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":214,"threads_started":1}
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
I20260926 20:52:29.012580 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:28.991811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:43759 (local address 127.8.249.129:33225)
0926 20:52:28.992640 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:28.992680 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:28.995125 (+ 2445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:28.995859 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:28.995873 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:28.996626 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:28.997435 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:28.997462 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:28.998461 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:28.998469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:29.000215 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:29.000227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:29.001310 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:29.001314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:29.001510 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:29.002208 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:29.002235 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:29.004467 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:29.008577 (+ 4110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:29.008585 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:29.008601 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:29.009060 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:29.009518 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:29.009521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:29.009523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:29.009706 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:29.010349 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:29.010355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:29.010697 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 20:52:29.011005 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 20:52:29.011418 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":358,"threads_started":1}
I20260926 20:52:29.014233 12988 heartbeater.cc:347] Connected to a master server at 127.8.249.190:43759
I20260926 20:52:29.014575 12988 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
I20260926 20:52:29.015532 12988 heartbeater.cc:511] Master 127.8.249.190:43759 requested a full tablet report, sending...
I20260926 20:52:29.017887 12799 ts_manager.cc:194] Registered new tserver with Master: e6183f252915475dbdcc5201f8e71cd2 (127.8.249.129:35679)
I20260926 20:52:29.020062 12799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:33225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:52:29.042083 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:43759
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:29.373288 12994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:29.373592 12994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:29.373684 12994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:29.380792 12994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:29.380909 12994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:29.381141 12994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:52:29.390872 12994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:43759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:29.392096 12994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:29.393419 12994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:29.406183 12999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:29.406219 13000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:29.406251 13002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:29.406970 12994 server_base.cc:1034] running on GCE node
I20260926 20:52:29.407584 12994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:29.408422 12994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:29.409633 12994 hybrid_clock.cc:648] HybridClock initialized: now 1790455949409597 us; error 51 us; skew 500 ppm
I20260926 20:52:29.409945 12994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:29 dist-test-slave-2dg3 krb5kdc[12754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455949, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:29.412974 12994 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:52:29.414307 12994 webserver.cc:460] Webserver started at http://127.8.249.130:36311/ using document root <none> and password file <none>
I20260926 20:52:29.415028 12994 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:29.415167 12994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:29.415478 12994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:29.418215 12994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "4076b2399ecd4cf9a4ee6846085d21cd"
format_stamp: "Formatted at 2026-09-26 20:52:29 on dist-test-slave-2dg3"
server_key: "91b6f2188e2a640dbd8ecdf59882c901"
server_key_iv: "ca1d9e7a1f52bb561456df94daee576f"
server_key_version: "encryptionkey@0"
I20260926 20:52:29.419041 12994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "4076b2399ecd4cf9a4ee6846085d21cd"
format_stamp: "Formatted at 2026-09-26 20:52:29 on dist-test-slave-2dg3"
server_key: "91b6f2188e2a640dbd8ecdf59882c901"
server_key_iv: "ca1d9e7a1f52bb561456df94daee576f"
server_key_version: "encryptionkey@0"
I20260926 20:52:29.424221 12994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:52:29.427857 13009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:29.429226 12994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:29.429415 12994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "4076b2399ecd4cf9a4ee6846085d21cd"
format_stamp: "Formatted at 2026-09-26 20:52:29 on dist-test-slave-2dg3"
server_key: "91b6f2188e2a640dbd8ecdf59882c901"
server_key_iv: "ca1d9e7a1f52bb561456df94daee576f"
server_key_version: "encryptionkey@0"
I20260926 20:52:29.429623 12994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:29.474087 12994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:29.478772 12994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:29.479138 12994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:29.480141 12994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:29.481528 12994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:29.481599 12994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:29.481755 12994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:29.481808 12994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:29.513182 12994 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:39707
I20260926 20:52:29.513276 13121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:39707 every 8 connection(s)
I20260926 20:52:29.514621 12994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:52:29.523136 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 12994
I20260926 20:52:29.523339 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:52:29.523839 9190 external_mini_cluster.cc:1428] Setting key bb9cd832a4004e2797a4e7dfb2a8e32b
Sep 26 20:52:29 dist-test-slave-2dg3 krb5kdc[12754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455949, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:29.537043 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:29.517139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:54433 (local address 127.8.249.190:43759)
0926 20:52:29.517567 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:29.517573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:29.518643 (+ 1070us) server_negotiation.cc:407] Connection header received
0926 20:52:29.520441 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:29.520444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:29.520513 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:29.520616 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:29.522056 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:29.522857 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:29.524979 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:29.525441 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:29.525563 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:29.529401 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:29.529410 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:29.529417 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:29.529473 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:29.533252 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:29.534404 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:29.534412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:29.534418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:29.534570 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:29.535150 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:29.535154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:29.535156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:29.535487 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:29.535615 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:29.536515 (+ 900us) server_negotiation.cc:299] Negotiation successful
0926 20:52:29.536682 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":194,"threads_started":1}
I20260926 20:52:29.538549 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:29.517733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:43759 (local address 127.8.249.130:54433)
0926 20:52:29.518484 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:29.518529 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:29.520209 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:29.520840 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:29.520847 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:29.521341 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:29.521879 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:29.521908 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:29.523015 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:29.523020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:29.524805 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:29.524818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:29.525693 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:29.525701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:29.525996 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:29.526672 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:29.526693 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:29.529173 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:29.533532 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:29.533545 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:29.533579 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:29.534251 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:29.534696 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:29.534704 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:29.534711 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:29.534999 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:29.535625 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:29.535636 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:29.536124 (+ 488us) client_negotiation.cc:769] Sending connection context
0926 20:52:29.536435 (+ 311us) client_negotiation.cc:240] Negotiation successful
0926 20:52:29.537049 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":327,"threads_started":1}
I20260926 20:52:29.539976 13122 heartbeater.cc:347] Connected to a master server at 127.8.249.190:43759
I20260926 20:52:29.540292 13122 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:29.541019 13122 heartbeater.cc:511] Master 127.8.249.190:43759 requested a full tablet report, sending...
I20260926 20:52:29.542374 12799 ts_manager.cc:194] Registered new tserver with Master: 4076b2399ecd4cf9a4ee6846085d21cd (127.8.249.130:39707)
I20260926 20:52:29.543268 12799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:54433
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:52:29.570139 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:43759
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:29.900312 13128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:29.900615 13128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:29.900789 13128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:29.908238 13128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:29.908401 13128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:29.908591 13128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:52:29.919183 13128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:43759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:29.920501 13128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:29.921821 13128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:29.935251 13136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:29.935283 13134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:29.935389 13133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:29.936261 13128 server_base.cc:1034] running on GCE node
I20260926 20:52:29.936898 13128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:29.937839 13128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:29.939026 13128 hybrid_clock.cc:648] HybridClock initialized: now 1790455949938965 us; error 60 us; skew 500 ppm
I20260926 20:52:29.939324 13128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:29 dist-test-slave-2dg3 krb5kdc[12754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455949, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:29.942312 13128 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:52:29.943831 13128 webserver.cc:460] Webserver started at http://127.8.249.131:41783/ using document root <none> and password file <none>
I20260926 20:52:29.944519 13128 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:29.944661 13128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:29.944970 13128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:29.948030 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "b83ee1e52548437ea8c1fd6bf44fe4d3"
format_stamp: "Formatted at 2026-09-26 20:52:29 on dist-test-slave-2dg3"
server_key: "66155d08e7ef191eaf6a2aacb99c4538"
server_key_iv: "5536841357d7e26e4bb5e80ffa28faa8"
server_key_version: "encryptionkey@0"
I20260926 20:52:29.948827 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "b83ee1e52548437ea8c1fd6bf44fe4d3"
format_stamp: "Formatted at 2026-09-26 20:52:29 on dist-test-slave-2dg3"
server_key: "66155d08e7ef191eaf6a2aacb99c4538"
server_key_iv: "5536841357d7e26e4bb5e80ffa28faa8"
server_key_version: "encryptionkey@0"
I20260926 20:52:29.953976 13128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 20:52:29.957504 13143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:29.958858 13128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:52:29.959023 13128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "b83ee1e52548437ea8c1fd6bf44fe4d3"
format_stamp: "Formatted at 2026-09-26 20:52:29 on dist-test-slave-2dg3"
server_key: "66155d08e7ef191eaf6a2aacb99c4538"
server_key_iv: "5536841357d7e26e4bb5e80ffa28faa8"
server_key_version: "encryptionkey@0"
I20260926 20:52:29.959225 13128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:29.981218 13128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:29.985441 13128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:29.985754 13128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:29.986922 13128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:29.988380 13128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:29.988492 13128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:29.988840 13128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:29.988933 13128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:30.021363 13128 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:44695
I20260926 20:52:30.021447 13255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:44695 every 8 connection(s)
I20260926 20:52:30.022631 13128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:52:30.024025 12988 heartbeater.cc:503] Master 127.8.249.190:43759 was elected leader, sending a full tablet report...
I20260926 20:52:30.030061 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 13128
I20260926 20:52:30.030289 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:52:30.030795 9190 external_mini_cluster.cc:1428] Setting key 4c3f7722cdc533348540008693b66f12
Sep 26 20:52:30 dist-test-slave-2dg3 krb5kdc[12754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455949, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:30.043872 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:30.025281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:52505 (local address 127.8.249.190:43759)
0926 20:52:30.025462 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:30.025468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:30.026816 (+ 1348us) server_negotiation.cc:407] Connection header received
0926 20:52:30.028805 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:30.028809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:30.028902 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:30.029007 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:30.030589 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.031677 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.033976 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.034672 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.034849 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:30.038703 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:30.038707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:30.038710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:30.038742 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:30.041113 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:30.041815 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:30.041819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:30.041821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:30.041932 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:30.042297 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:30.042300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:30.042302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:30.042549 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:30.042660 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:30.043433 (+ 773us) server_negotiation.cc:299] Negotiation successful
0926 20:52:30.043597 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:52:30.045053 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:30.025911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:43759 (local address 127.8.249.131:52505)
0926 20:52:30.026649 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:30.026685 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:30.028490 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:30.029220 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:30.029228 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:30.029759 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:30.030407 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:30.030437 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.031866 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.031871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:30.033810 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:30.033829 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.035179 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.035187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:30.035497 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:30.036228 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:30.036251 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:30.038504 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:30.041247 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:30.041254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:30.041267 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:30.041707 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:30.042023 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:30.042025 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:30.042028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:30.042196 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:30.042684 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:30.042690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:30.043111 (+ 421us) client_negotiation.cc:769] Sending connection context
0926 20:52:30.043368 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 20:52:30.043761 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":261,"threads_started":1}
I20260926 20:52:30.046638 13256 heartbeater.cc:347] Connected to a master server at 127.8.249.190:43759
I20260926 20:52:30.046998 13256 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:30.047729 13256 heartbeater.cc:511] Master 127.8.249.190:43759 requested a full tablet report, sending...
I20260926 20:52:30.049114 12798 ts_manager.cc:194] Registered new tserver with Master: b83ee1e52548437ea8c1fd6bf44fe4d3 (127.8.249.131:44695)
I20260926 20:52:30.050000 12798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:52505
I20260926 20:52:30.058487 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:52:30 dist-test-slave-2dg3 krb5kdc[12754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:52:30 dist-test-slave-2dg3 krb5kdc[12754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:52:30.106433 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:30.092280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.8.249.129:35679)
0926 20:52:30.093020 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:30.093028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:30.093055 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:52:30.093116 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:30.093123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:30.093361 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:30.093499 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:30.094919 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.095934 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.097077 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.097615 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.097709 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:30.101381 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:30.101392 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:30.101407 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:30.101444 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:30.104251 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:30.104841 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:30.104847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:30.104849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:30.104967 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:30.105257 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:30.105260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:30.105262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:30.105561 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:30.105780 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:30.105972 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 20:52:30.106163 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":422,"threads_started":1}
Sep 26 20:52:30 dist-test-slave-2dg3 krb5kdc[12754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:30.123484 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:30.110516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.8.249.190:43759)
0926 20:52:30.110711 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:30.110718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:30.110942 (+ 224us) server_negotiation.cc:407] Connection header received
0926 20:52:30.111088 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:30.111092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:30.111151 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:30.111228 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:30.112318 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.113004 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.114154 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.114539 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.114626 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:30.118615 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:30.118620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:30.118623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:30.118655 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:30.121210 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:30.121766 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:30.121769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:30.121771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:30.121881 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:30.122167 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:30.122170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:30.122172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:30.122460 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:30.122642 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:30.122943 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 20:52:30.123098 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 20:52:30.124145 12798 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50270
I20260926 20:52:30.125177 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 12860
I20260926 20:52:30.137636 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:30.125991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.8.249.129:35679)
0926 20:52:30.126146 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:30.126151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:30.126415 (+ 264us) server_negotiation.cc:407] Connection header received
0926 20:52:30.126605 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:30.126609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:30.126671 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:30.126756 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:30.128033 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.128736 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.129906 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.130340 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.130437 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:30.132606 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:30.132611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:30.132614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:30.132647 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:30.135379 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:30.136047 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:30.136051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:30.136053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:30.136183 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:30.136548 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:30.136551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:30.136553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:30.136795 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:30.136967 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:30.137242 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 20:52:30.137403 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:52:30.138291 12983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:30.219466 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 12860
I20260926 20:52:30.235375 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 12994
Sep 26 20:52:30 dist-test-slave-2dg3 krb5kdc[12754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:52:30.250607 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:30.236312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.8.249.130:39707)
0926 20:52:30.236876 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:30.236883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:30.236910 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:52:30.236970 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:30.236976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:30.237189 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:30.237321 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:30.238381 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.239433 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.240610 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.241158 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.241255 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:30.245355 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:30.245366 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:30.245381 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:30.245426 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:30.248195 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:30.248849 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:30.248856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:30.248858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:30.248989 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:30.249304 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:30.249307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:30.249309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:30.249634 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:30.249845 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:30.250118 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 20:52:30.250304 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":278,"threads_started":1}
I20260926 20:52:30.251884 13117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:30.334908 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 12994
I20260926 20:52:30.350674 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 13128
Sep 26 20:52:30 dist-test-slave-2dg3 krb5kdc[12754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:52:30.367520 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:30.351726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.8.249.131:44695)
0926 20:52:30.352377 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:30.352383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:30.352421 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:52:30.352480 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:30.352486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:30.352694 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:30.352816 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:30.354027 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.355238 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.356571 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:30.357120 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:30.357214 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:30.361536 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:30.361549 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:30.361563 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:30.361605 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:30.364984 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:30.365683 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:30.365690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:30.365692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:30.365814 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:30.366126 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:30.366129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:30.366131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:30.366469 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:30.366695 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:30.367007 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 20:52:30.367214 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":341,"threads_started":1}
I20260926 20:52:30.368798 13251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:30.451169 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 13128
I20260926 20:52:30.467228 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 12767
I20260926 20:52:30.468415 12829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:30.569824 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 12767
2026-09-26T20:52:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3864 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:52:30 dist-test-slave-2dg3 krb5kdc[13275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:52:30 dist-test-slave-2dg3 krb5kdc[13275](info): set up 2 sockets
Sep 26 20:52:30 dist-test-slave-2dg3 krb5kdc[13275](info): commencing operation
krb5kdc: starting...
W20260926 20:52:31.671387 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:52:31 dist-test-slave-2dg3 krb5kdc[13275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:52:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:52:31.875422 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:44463
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:44463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:32.199940 13288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:32.200258 13288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:32.200383 13288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:32.207854 13288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:32.207988 13288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:32.208073 13288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:32.208173 13288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:32.208272 13288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:32.218356 13288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:44463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:44463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:32.219731 13288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:32.221035 13288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:32.232765 13294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:32.232846 13293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:32.232869 13296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:32.233868 13288 server_base.cc:1034] running on GCE node
I20260926 20:52:32.234709 13288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:32.236281 13288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:32.237490 13288 hybrid_clock.cc:648] HybridClock initialized: now 1790455952237429 us; error 55 us; skew 500 ppm
I20260926 20:52:32.237857 13288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:32 dist-test-slave-2dg3 krb5kdc[13275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455952, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:32.241107 13288 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:52:32.242331 13288 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:32.242455 13288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:32.242875 13288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:32.245972 13288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "486c9110d3c947468fff9f6601faf50e"
format_stamp: "Formatted at 2026-09-26 20:52:32 on dist-test-slave-2dg3"
server_key: "448614e5e9fdb832fd26a969a9b1a349"
server_key_iv: "e672c5803a2497b65ca86d0a938b9e03"
server_key_version: "encryptionkey@0"
I20260926 20:52:32.246876 13288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "486c9110d3c947468fff9f6601faf50e"
format_stamp: "Formatted at 2026-09-26 20:52:32 on dist-test-slave-2dg3"
server_key: "448614e5e9fdb832fd26a969a9b1a349"
server_key_iv: "e672c5803a2497b65ca86d0a938b9e03"
server_key_version: "encryptionkey@0"
I20260926 20:52:32.252043 13288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 20:52:32.255700 13302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:32.256946 13288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:52:32.257095 13288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "486c9110d3c947468fff9f6601faf50e"
format_stamp: "Formatted at 2026-09-26 20:52:32 on dist-test-slave-2dg3"
server_key: "448614e5e9fdb832fd26a969a9b1a349"
server_key_iv: "e672c5803a2497b65ca86d0a938b9e03"
server_key_version: "encryptionkey@0"
I20260926 20:52:32.257364 13288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:32.285570 13288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:32.289674 13288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:32.289971 13288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:32.307919 13288 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:44463
I20260926 20:52:32.307943 13353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:44463 every 8 connection(s)
I20260926 20:52:32.308800 13288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:32.312327 13354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:32.314687 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 13288
I20260926 20:52:32.314970 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:32.315595 9190 external_mini_cluster.cc:1428] Setting key 6eac3ecfc3d79218d70c8343839b8963
I20260926 20:52:32.324193 13354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e: Bootstrap starting.
I20260926 20:52:32.328189 13354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:32.329550 13354 log.cc:824] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:52:32 dist-test-slave-2dg3 krb5kdc[13275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:32.332556 13354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e: No bootstrap required, opened a new log
I20260926 20:52:32.340138 13357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:32.317517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.8.249.190:44463)
0926 20:52:32.318265 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:32.318277 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:32.318318 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:52:32.319970 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:32.319993 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:32.320419 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:32.320760 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:32.322718 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:32.324320 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:32.326240 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:32.327056 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:32.327223 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:32.332129 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:32.332138 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:32.332153 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:32.332195 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:32.335571 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:32.336370 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:32.336385 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:32.336393 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:32.336577 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:32.337020 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:32.337027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:32.337031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:32.337650 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:32.338030 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:32.338497 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 20:52:32.339009 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":358,"threads_started":1}
I20260926 20:52:32.340613 13354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486c9110d3c947468fff9f6601faf50e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44463 } }
I20260926 20:52:32.341027 13354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:32.341141 13354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486c9110d3c947468fff9f6601faf50e, State: Initialized, Role: FOLLOWER
I20260926 20:52:32.341799 13354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486c9110d3c947468fff9f6601faf50e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44463 } }
I20260926 20:52:32.342103 13354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:32.342245 13354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:32.342420 13354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:32.344964 13354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486c9110d3c947468fff9f6601faf50e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44463 } }
I20260926 20:52:32.345402 13354 leader_election.cc:302] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486c9110d3c947468fff9f6601faf50e; no voters:
I20260926 20:52:32.346001 13354 leader_election.cc:288] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:32.346175 13359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:32.346971 13359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [term 1 LEADER]: Becoming Leader. State: Replica: 486c9110d3c947468fff9f6601faf50e, State: Running, Role: LEADER
I20260926 20:52:32.347694 13359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486c9110d3c947468fff9f6601faf50e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44463 } }
I20260926 20:52:32.348332 13354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:32.351076 13361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486c9110d3c947468fff9f6601faf50e. Latest consensus state: current_term: 1 leader_uuid: "486c9110d3c947468fff9f6601faf50e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486c9110d3c947468fff9f6601faf50e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44463 } } }
I20260926 20:52:32.351081 13360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "486c9110d3c947468fff9f6601faf50e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486c9110d3c947468fff9f6601faf50e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44463 } } }
I20260926 20:52:32.351572 13361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:32.351568 13360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:32.352507 13367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:32.357414 13367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:32.366273 13367 catalog_manager.cc:1347] Generated new cluster ID: 775e04553dff4ebbbd6089f33dce3a07
I20260926 20:52:32.366384 13367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:32.379925 13367 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:32.381795 13367 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:32.392954 13367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e: Generated new TSK 0
I20260926 20:52:32.393788 13367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:52:32.444491 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44463
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:32.771893 13380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:32.772284 13380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:32.772477 13380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:32.779758 13380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:32.779884 13380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:32.780081 13380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:52:32.789826 13380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:32.791221 13380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:32.792613 13380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:32.805892 13385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:32.805852 13386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:32.805917 13388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:32.806650 13380 server_base.cc:1034] running on GCE node
I20260926 20:52:32.807286 13380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:32.808212 13380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:32.809454 13380 hybrid_clock.cc:648] HybridClock initialized: now 1790455952809394 us; error 50 us; skew 500 ppm
I20260926 20:52:32.809770 13380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:32 dist-test-slave-2dg3 krb5kdc[13275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455952, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:32.812814 13380 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:52:32.813994 13380 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:32.814128 13380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:32.814443 13380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:32.817392 13380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "e43eeb2448b74afeaa80927b9e85ade4"
format_stamp: "Formatted at 2026-09-26 20:52:32 on dist-test-slave-2dg3"
server_key: "4838c2a83bf8ccb977b56c2c89dd54ea"
server_key_iv: "3db34014b93dc41a8c8850a5b9153b92"
server_key_version: "encryptionkey@0"
I20260926 20:52:32.818199 13380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "e43eeb2448b74afeaa80927b9e85ade4"
format_stamp: "Formatted at 2026-09-26 20:52:32 on dist-test-slave-2dg3"
server_key: "4838c2a83bf8ccb977b56c2c89dd54ea"
server_key_iv: "3db34014b93dc41a8c8850a5b9153b92"
server_key_version: "encryptionkey@0"
I20260926 20:52:32.823146 13380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 20:52:32.826414 13394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:32.827749 13380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:32.827915 13380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "e43eeb2448b74afeaa80927b9e85ade4"
format_stamp: "Formatted at 2026-09-26 20:52:32 on dist-test-slave-2dg3"
server_key: "4838c2a83bf8ccb977b56c2c89dd54ea"
server_key_iv: "3db34014b93dc41a8c8850a5b9153b92"
server_key_version: "encryptionkey@0"
I20260926 20:52:32.828132 13380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:32.854019 13380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:32.858078 13380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:32.858392 13380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:32.859386 13380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:32.860687 13380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:32.860811 13380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:32.860944 13380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:32.861030 13380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:32.894626 13380 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:43721
I20260926 20:52:32.894712 13506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:43721 every 8 connection(s)
I20260926 20:52:32.895602 13380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:52:32.904711 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 13380
I20260926 20:52:32.904934 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:52:32.905687 9190 external_mini_cluster.cc:1428] Setting key 6212e88211d2e6935d9f4606a3f77ec0
Sep 26 20:52:32 dist-test-slave-2dg3 krb5kdc[13275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455952, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:32.916314 13509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:32.897910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:33311 (local address 127.8.249.190:44463)
0926 20:52:32.898334 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:32.898339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:32.899241 (+ 902us) server_negotiation.cc:407] Connection header received
0926 20:52:32.901041 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:32.901046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:32.901131 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:32.901246 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:32.902724 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:32.903492 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:32.905565 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:32.906044 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:32.906151 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:32.909952 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:32.909957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:32.909960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:32.909992 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:32.913137 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:32.913931 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:32.913938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:32.913942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:32.914064 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:32.914488 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:32.914496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:32.914499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:32.914779 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:32.914984 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:32.915719 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 20:52:32.915945 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":233,"threads_started":1}
I20260926 20:52:32.917362 13510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:32.898370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44463 (local address 127.8.249.129:33311)
0926 20:52:32.899086 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:32.899129 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:32.900755 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:32.901479 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:32.901487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:32.902001 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:32.902553 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:32.902570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:32.903663 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:32.903667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:32.905405 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:32.905421 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:32.906291 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:32.906297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:32.906496 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:32.907201 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:32.907230 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:32.909710 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:32.913305 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:32.913314 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:32.913334 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:32.913800 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:32.914180 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:32.914184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:32.914188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:32.914377 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:32.914958 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:32.914966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:32.915341 (+ 375us) client_negotiation.cc:769] Sending connection context
0926 20:52:32.915622 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 20:52:32.916203 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":280,"threads_started":1}
I20260926 20:52:32.919137 13507 heartbeater.cc:347] Connected to a master server at 127.8.249.190:44463
I20260926 20:52:32.919466 13507 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:32.920136 13507 heartbeater.cc:511] Master 127.8.249.190:44463 requested a full tablet report, sending...
I20260926 20:52:32.922627 13319 ts_manager.cc:194] Registered new tserver with Master: e43eeb2448b74afeaa80927b9e85ade4 (127.8.249.129:43721)
I20260926 20:52:32.924872 13319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:33311
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:52:32.952378 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44463
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:33.276378 13513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:33.276678 13513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:33.276780 13513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:33.284229 13513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:33.284341 13513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:33.284457 13513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:52:33.294344 13513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:33.295609 13513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:33.296820 13513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:33.309598 13519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:33.309832 13521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:33.310257 13518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:33.310961 13513 server_base.cc:1034] running on GCE node
I20260926 20:52:33.311568 13513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:33.312574 13513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:33.313846 13513 hybrid_clock.cc:648] HybridClock initialized: now 1790455953313772 us; error 50 us; skew 500 ppm
I20260926 20:52:33.314186 13513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:33 dist-test-slave-2dg3 krb5kdc[13275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455953, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:33.317313 13513 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:52:33.318503 13513 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:33.318591 13513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:33.318977 13513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:33.321771 13513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "305c57b0b70246b6a39f3b05e4a5e09b"
format_stamp: "Formatted at 2026-09-26 20:52:33 on dist-test-slave-2dg3"
server_key: "46be200f0ad5669926575771de6ed523"
server_key_iv: "eb9ea1884c617fbfc127c1ae72304fc7"
server_key_version: "encryptionkey@0"
I20260926 20:52:33.322548 13513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "305c57b0b70246b6a39f3b05e4a5e09b"
format_stamp: "Formatted at 2026-09-26 20:52:33 on dist-test-slave-2dg3"
server_key: "46be200f0ad5669926575771de6ed523"
server_key_iv: "eb9ea1884c617fbfc127c1ae72304fc7"
server_key_version: "encryptionkey@0"
I20260926 20:52:33.327606 13513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 20:52:33.330891 13527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:33.332240 13513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:52:33.332407 13513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "305c57b0b70246b6a39f3b05e4a5e09b"
format_stamp: "Formatted at 2026-09-26 20:52:33 on dist-test-slave-2dg3"
server_key: "46be200f0ad5669926575771de6ed523"
server_key_iv: "eb9ea1884c617fbfc127c1ae72304fc7"
server_key_version: "encryptionkey@0"
I20260926 20:52:33.332623 13513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:33.352782 13513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:33.357035 13513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:33.357312 13513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:33.358280 13513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:33.359655 13513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:33.359779 13513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:33.359901 13513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:33.359993 13513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:33.392542 13513 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:38677
I20260926 20:52:33.392598 13639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:38677 every 8 connection(s)
I20260926 20:52:33.393438 13513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:52:33.402359 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 13513
I20260926 20:52:33.402570 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:52:33.403146 9190 external_mini_cluster.cc:1428] Setting key 6c940a2520ff4cb30c7d7d5bf444ff09
Sep 26 20:52:33 dist-test-slave-2dg3 krb5kdc[13275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455953, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:33.415179 13509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:33.395915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:41639 (local address 127.8.249.190:44463)
0926 20:52:33.396089 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:33.396094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:33.397317 (+ 1223us) server_negotiation.cc:407] Connection header received
0926 20:52:33.399119 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:33.399124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:33.399192 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:33.399281 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:33.401007 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.401697 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.404022 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.404519 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.404665 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:33.408885 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:33.408890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:33.408893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:33.408933 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:33.411625 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:33.412364 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:33.412373 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:33.412379 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:33.412535 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:33.413022 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:33.413028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:33.413031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:33.413402 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:33.413542 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:33.414448 (+ 906us) server_negotiation.cc:299] Negotiation successful
0926 20:52:33.414701 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:52:33.415834 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:33.396380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44463 (local address 127.8.249.130:41639)
0926 20:52:33.397126 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:33.397175 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:33.398848 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:33.399478 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:33.399487 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:33.400208 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:33.400826 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:33.400844 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.401842 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.401846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:33.403869 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:33.403882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.404761 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.404769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:33.404981 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:33.405708 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:33.405751 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:33.408635 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:33.411751 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:33.411758 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:33.411772 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:33.412219 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:33.412642 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:33.412645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:33.412647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:33.412861 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:33.413633 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:33.413641 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:33.414042 (+ 401us) client_negotiation.cc:769] Sending connection context
0926 20:52:33.414286 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 20:52:33.414676 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":278,"threads_started":1}
I20260926 20:52:33.417321 13640 heartbeater.cc:347] Connected to a master server at 127.8.249.190:44463
I20260926 20:52:33.417626 13640 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:33.418255 13640 heartbeater.cc:511] Master 127.8.249.190:44463 requested a full tablet report, sending...
I20260926 20:52:33.419669 13319 ts_manager.cc:194] Registered new tserver with Master: 305c57b0b70246b6a39f3b05e4a5e09b (127.8.249.130:38677)
I20260926 20:52:33.420465 13319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:41639
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:52:33.448551 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44463
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:33.777112 13645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:33.777392 13645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:33.777487 13645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:33.784739 13645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:33.784847 13645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:33.784962 13645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:52:33.795092 13645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:33.796406 13645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:33.797806 13645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:33.810837 13653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:33.810905 13651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:33.810976 13650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:33.811765 13645 server_base.cc:1034] running on GCE node
I20260926 20:52:33.812395 13645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:33.813344 13645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:33.814570 13645 hybrid_clock.cc:648] HybridClock initialized: now 1790455953814518 us; error 66 us; skew 500 ppm
I20260926 20:52:33.814939 13645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:33 dist-test-slave-2dg3 krb5kdc[13275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455953, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:33.818192 13645 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:52:33.819463 13645 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:33.819593 13645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:33.819900 13645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:33.822763 13645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "4b90783690f14d5686e4c02394ca02fd"
format_stamp: "Formatted at 2026-09-26 20:52:33 on dist-test-slave-2dg3"
server_key: "da542548dabe18b78c5ba333c6a56c84"
server_key_iv: "e952b02905a5eb7d14ce8e6add52f721"
server_key_version: "encryptionkey@0"
I20260926 20:52:33.823629 13645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "4b90783690f14d5686e4c02394ca02fd"
format_stamp: "Formatted at 2026-09-26 20:52:33 on dist-test-slave-2dg3"
server_key: "da542548dabe18b78c5ba333c6a56c84"
server_key_iv: "e952b02905a5eb7d14ce8e6add52f721"
server_key_version: "encryptionkey@0"
I20260926 20:52:33.828513 13645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:52:33.832226 13659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:33.833581 13645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:52:33.833709 13645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "4b90783690f14d5686e4c02394ca02fd"
format_stamp: "Formatted at 2026-09-26 20:52:33 on dist-test-slave-2dg3"
server_key: "da542548dabe18b78c5ba333c6a56c84"
server_key_iv: "e952b02905a5eb7d14ce8e6add52f721"
server_key_version: "encryptionkey@0"
I20260926 20:52:33.833889 13645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:33.858875 13645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:33.863260 13645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:33.863487 13645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:33.864365 13645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:33.865604 13645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:33.865686 13645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:33.865756 13645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:33.865809 13645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:33.899571 13645 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:37975
I20260926 20:52:33.899606 13771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:37975 every 8 connection(s)
I20260926 20:52:33.900382 13645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:52:33.908981 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 13645
I20260926 20:52:33.909193 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:52:33.909601 9190 external_mini_cluster.cc:1428] Setting key f07e0f62f094329da6718919ec8f46ae
Sep 26 20:52:33 dist-test-slave-2dg3 krb5kdc[13275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455953, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:33.919546 13509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:33.902859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:60965 (local address 127.8.249.190:44463)
0926 20:52:33.903033 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:33.903038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:33.904091 (+ 1053us) server_negotiation.cc:407] Connection header received
0926 20:52:33.905936 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:33.905945 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:33.906029 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:33.906144 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:33.907700 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.908444 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.910090 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.910695 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.910839 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:33.914469 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:33.914473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:33.914477 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:33.914507 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:33.916980 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:33.917624 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:33.917628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:33.917630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:33.917743 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:33.918095 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:33.918098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:33.918100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:33.918345 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:33.918451 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:33.919154 (+ 703us) server_negotiation.cc:299] Negotiation successful
0926 20:52:33.919311 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:52:33.920531 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:33.903259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44463 (local address 127.8.249.131:60965)
0926 20:52:33.903933 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:33.903968 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:33.905621 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:33.906366 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:33.906374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:33.906939 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:33.907527 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:33.907544 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.908603 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.908607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:33.909959 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:33.909970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.910954 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.910963 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:33.911205 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:33.911962 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:33.911989 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:33.914285 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:33.917103 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:33.917109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:33.917121 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:33.917521 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:33.917830 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:33.917833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:33.917836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:33.918002 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:33.918445 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:33.918451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:33.918755 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 20:52:33.919051 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 20:52:33.919441 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":278,"threads_started":1}
I20260926 20:52:33.921865 13772 heartbeater.cc:347] Connected to a master server at 127.8.249.190:44463
I20260926 20:52:33.922158 13772 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:33.922765 13772 heartbeater.cc:511] Master 127.8.249.190:44463 requested a full tablet report, sending...
I20260926 20:52:33.924400 13319 ts_manager.cc:194] Registered new tserver with Master: 4b90783690f14d5686e4c02394ca02fd (127.8.249.131:37975)
I20260926 20:52:33.925134 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:52:33.925365 13319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:60965
I20260926 20:52:33.928186 13507 heartbeater.cc:503] Master 127.8.249.190:44463 was elected leader, sending a full tablet report...
I20260926 20:52:33.942531 13509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:33.929704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42100 (local address 127.8.249.190:44463)
0926 20:52:33.929873 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:33.929877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:33.930412 (+ 535us) server_negotiation.cc:407] Connection header received
0926 20:52:33.930605 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:33.930609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:33.930681 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:33.930781 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:33.932442 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.933158 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.934420 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.934921 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.935058 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:33.937397 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:33.937403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:33.937409 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:33.937453 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:33.939922 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:33.940578 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:33.940585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:33.940589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:33.940722 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:33.941084 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:33.941092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:33.941096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:33.941430 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:33.941686 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:33.942003 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 20:52:33.942209 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:52:33.946288 13319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:52:33.953184 13319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:52:33.987496 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:33.964836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.8.249.130:38677)
0926 20:52:33.977752 (+ 12916us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:33.977765 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:33.977798 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:52:33.977919 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:33.977943 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:33.978244 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:33.978455 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:33.979544 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.981365 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.984378 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.985892 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.986030 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:33.986246 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:33.986975 (+ 729us) server_negotiation.cc:299] Negotiation successful
0926 20:52:33.987152 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12724,"thread_start_us":12583,"threads_started":1}
I20260926 20:52:33.987607 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:33.964646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:38677 (local address 127.0.0.1:35822)
0926 20:52:33.976869 (+ 12223us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:33.976910 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:33.977098 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:33.978754 (+ 1656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:33.978766 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:33.978824 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:33.979301 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:33.979317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.981560 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.981569 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:33.984231 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:33.984243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.986397 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.986404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:33.986662 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:33.986722 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 20:52:33.986877 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 20:52:33.987103 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11936,"mutex_wait_us":1,"thread_start_us":16197,"threads_started":1}
I20260926 20:52:33.990774 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:33.964719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:43721 (local address 127.0.0.1:42064)
0926 20:52:33.981115 (+ 16396us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:33.981139 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:33.981270 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:33.983466 (+ 2196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:33.983471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:33.983497 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:33.984065 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:33.984083 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.985952 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.985956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:33.988126 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:33.988139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.989959 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.989965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:33.990120 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:33.990151 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 20:52:33.990250 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 20:52:33.990376 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16270,"mutex_wait_us":128,"thread_start_us":11734,"threads_started":1}
I20260926 20:52:33.990861 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:33.965337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.8.249.129:43721)
0926 20:52:33.982835 (+ 17498us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:33.982852 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:33.982881 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:52:33.982983 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:33.982991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:33.983240 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:33.983470 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:33.984269 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.985464 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.988289 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.989725 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.989865 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:33.990105 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:33.990307 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 20:52:33.990490 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17313,"thread_start_us":17184,"threads_started":1}
I20260926 20:52:33.991858 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:33.965109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:37975 (local address 127.0.0.1:50098)
0926 20:52:33.982160 (+ 17051us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:33.982177 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:33.982291 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:33.983742 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:33.983747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:33.983774 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:33.984282 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:33.984295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.986243 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.986248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:33.988739 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:33.988752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.990860 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.990866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:33.991236 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:33.991273 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:52:33.991382 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 20:52:33.991506 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16916,"mutex_wait_us":45,"thread_start_us":16762,"threads_started":1}
I20260926 20:52:33.991926 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:33.965852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.8.249.131:37975)
0926 20:52:33.981613 (+ 15761us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:33.981626 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:33.983071 (+ 1445us) server_negotiation.cc:407] Connection header received
0926 20:52:33.983165 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:33.983172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:33.983389 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:33.983563 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:33.984495 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.986083 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.988911 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:33.990349 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:33.990475 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:33.990677 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:33.991445 (+ 768us) server_negotiation.cc:299] Negotiation successful
0926 20:52:33.991608 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15526,"thread_start_us":15361,"threads_started":1}
I20260926 20:52:33.994462 13442 tablet_service.cc:1467] Processing CreateTablet for tablet 856fc8c732204949bb4cb350f3225be7 (DEFAULT_TABLE table=test-table [id=b9ef86d630cf45b39a152c941221c909]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:33.991691 13575 tablet_service.cc:1467] Processing CreateTablet for tablet 856fc8c732204949bb4cb350f3225be7 (DEFAULT_TABLE table=test-table [id=b9ef86d630cf45b39a152c941221c909]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:33.995285 13707 tablet_service.cc:1467] Processing CreateTablet for tablet 856fc8c732204949bb4cb350f3225be7 (DEFAULT_TABLE table=test-table [id=b9ef86d630cf45b39a152c941221c909]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:33.995987 13442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856fc8c732204949bb4cb350f3225be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:33.996256 13575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856fc8c732204949bb4cb350f3225be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:33.996527 13707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856fc8c732204949bb4cb350f3225be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:34.008054 13790 tablet_bootstrap.cc:492] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b: Bootstrap starting.
I20260926 20:52:34.010883 13791 tablet_bootstrap.cc:492] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4: Bootstrap starting.
I20260926 20:52:34.012460 13792 tablet_bootstrap.cc:492] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd: Bootstrap starting.
I20260926 20:52:34.012693 13790 tablet_bootstrap.cc:654] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:34.014110 13790 log.cc:824] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:34.016201 13792 tablet_bootstrap.cc:654] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:34.017503 13792 log.cc:824] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:34.017601 13790 tablet_bootstrap.cc:492] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b: No bootstrap required, opened a new log
I20260926 20:52:34.017649 13791 tablet_bootstrap.cc:654] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:34.017897 13790 ts_tablet_manager.cc:1397] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 20:52:34.019063 13791 log.cc:824] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:34.022171 13792 tablet_bootstrap.cc:492] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd: No bootstrap required, opened a new log
I20260926 20:52:34.022365 13791 tablet_bootstrap.cc:492] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4: No bootstrap required, opened a new log
I20260926 20:52:34.022542 13792 ts_tablet_manager.cc:1397] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260926 20:52:34.022651 13791 ts_tablet_manager.cc:1397] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20260926 20:52:34.029080 13790 raft_consensus.cc:348] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c57b0b70246b6a39f3b05e4a5e09b" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 38677 } } peers { permanent_uuid: "e43eeb2448b74afeaa80927b9e85ade4" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 43721 } } peers { permanent_uuid: "4b90783690f14d5686e4c02394ca02fd" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37975 } }
I20260926 20:52:34.029598 13790 raft_consensus.cc:374] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:34.029737 13790 raft_consensus.cc:729] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305c57b0b70246b6a39f3b05e4a5e09b, State: Initialized, Role: FOLLOWER
I20260926 20:52:34.030442 13790 consensus_queue.cc:260] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c57b0b70246b6a39f3b05e4a5e09b" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 38677 } } peers { permanent_uuid: "e43eeb2448b74afeaa80927b9e85ade4" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 43721 } } peers { permanent_uuid: "4b90783690f14d5686e4c02394ca02fd" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37975 } }
I20260926 20:52:34.030838 13792 raft_consensus.cc:348] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c57b0b70246b6a39f3b05e4a5e09b" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 38677 } } peers { permanent_uuid: "e43eeb2448b74afeaa80927b9e85ade4" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 43721 } } peers { permanent_uuid: "4b90783690f14d5686e4c02394ca02fd" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37975 } }
I20260926 20:52:34.031337 13792 raft_consensus.cc:374] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:34.031450 13792 raft_consensus.cc:729] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b90783690f14d5686e4c02394ca02fd, State: Initialized, Role: FOLLOWER
I20260926 20:52:34.032091 13792 consensus_queue.cc:260] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c57b0b70246b6a39f3b05e4a5e09b" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 38677 } } peers { permanent_uuid: "e43eeb2448b74afeaa80927b9e85ade4" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 43721 } } peers { permanent_uuid: "4b90783690f14d5686e4c02394ca02fd" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37975 } }
I20260926 20:52:34.033272 13772 heartbeater.cc:503] Master 127.8.249.190:44463 was elected leader, sending a full tablet report...
I20260926 20:52:34.034710 13791 raft_consensus.cc:348] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c57b0b70246b6a39f3b05e4a5e09b" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 38677 } } peers { permanent_uuid: "e43eeb2448b74afeaa80927b9e85ade4" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 43721 } } peers { permanent_uuid: "4b90783690f14d5686e4c02394ca02fd" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37975 } }
I20260926 20:52:34.035317 13791 raft_consensus.cc:374] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:34.035442 13791 raft_consensus.cc:729] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e43eeb2448b74afeaa80927b9e85ade4, State: Initialized, Role: FOLLOWER
I20260926 20:52:34.036316 13791 consensus_queue.cc:260] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c57b0b70246b6a39f3b05e4a5e09b" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 38677 } } peers { permanent_uuid: "e43eeb2448b74afeaa80927b9e85ade4" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 43721 } } peers { permanent_uuid: "4b90783690f14d5686e4c02394ca02fd" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37975 } }
I20260926 20:52:34.038025 13792 ts_tablet_manager.cc:1428] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd: Time spent starting tablet: real 0.015s user 0.002s sys 0.010s
I20260926 20:52:34.038836 13790 ts_tablet_manager.cc:1428] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b: Time spent starting tablet: real 0.021s user 0.019s sys 0.000s
I20260926 20:52:34.039444 13640 heartbeater.cc:503] Master 127.8.249.190:44463 was elected leader, sending a full tablet report...
I20260926 20:52:34.040692 13791 ts_tablet_manager.cc:1428] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260926 20:52:34.123540 13797 raft_consensus.cc:482] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:34.123814 13797 raft_consensus.cc:504] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c57b0b70246b6a39f3b05e4a5e09b" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 38677 } } peers { permanent_uuid: "e43eeb2448b74afeaa80927b9e85ade4" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 43721 } } peers { permanent_uuid: "4b90783690f14d5686e4c02394ca02fd" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37975 } }
I20260926 20:52:34.125469 13797 leader_election.cc:288] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 305c57b0b70246b6a39f3b05e4a5e09b (127.8.249.130:38677), e43eeb2448b74afeaa80927b9e85ade4 (127.8.249.129:43721)
I20260926 20:52:34.131953 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.126066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:51181 (local address 127.8.249.129:43721)
0926 20:52:34.126266 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:34.126272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:34.126443 (+ 171us) server_negotiation.cc:407] Connection header received
0926 20:52:34.126710 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:34.126715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:34.126827 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:34.126939 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:34.127330 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.128302 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.129995 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.130982 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.131108 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:34.131161 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:34.131536 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 20:52:34.131681 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:52:34.131956 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.125756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:43721 (local address 127.8.249.131:51181)
0926 20:52:34.126331 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:34.126347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:34.126529 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:34.126970 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:34.126975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:34.127002 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:34.127175 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:34.127185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.128433 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.128438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:34.129832 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:34.129844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.131135 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.131144 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:34.131349 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:34.131393 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:52:34.131511 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:52:34.131650 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402}
I20260926 20:52:34.132006 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.125820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:53513 (local address 127.8.249.130:38677)
0926 20:52:34.126045 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:34.126054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:34.126748 (+ 694us) server_negotiation.cc:407] Connection header received
0926 20:52:34.126985 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:34.126992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:34.127066 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:34.127151 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:34.127701 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.128527 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.130318 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.131110 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.131259 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:34.131333 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:34.131653 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 20:52:34.131780 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:52:34.132161 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.125861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:38677 (local address 127.8.249.131:53513)
0926 20:52:34.126639 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:34.126653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:34.126797 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:34.127181 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:34.127184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:34.127204 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:34.127539 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:34.127547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.128669 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.128673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:34.130164 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:34.130179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.131325 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.131330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:34.131502 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:34.131532 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 20:52:34.131623 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:52:34.131736 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":1,"thread_start_us":437,"threads_started":1}
I20260926 20:52:34.133131 13595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856fc8c732204949bb4cb350f3225be7" candidate_uuid: "4b90783690f14d5686e4c02394ca02fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305c57b0b70246b6a39f3b05e4a5e09b" is_pre_election: true
I20260926 20:52:34.133361 13462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856fc8c732204949bb4cb350f3225be7" candidate_uuid: "4b90783690f14d5686e4c02394ca02fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e43eeb2448b74afeaa80927b9e85ade4" is_pre_election: true
I20260926 20:52:34.133623 13595 raft_consensus.cc:2454] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b90783690f14d5686e4c02394ca02fd in term 0.
I20260926 20:52:34.133936 13462 raft_consensus.cc:2454] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b90783690f14d5686e4c02394ca02fd in term 0.
I20260926 20:52:34.134311 13660 leader_election.cc:302] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 305c57b0b70246b6a39f3b05e4a5e09b, 4b90783690f14d5686e4c02394ca02fd; no voters:
I20260926 20:52:34.134665 13797 raft_consensus.cc:2781] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:52:34.134830 13797 raft_consensus.cc:482] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:34.134956 13797 raft_consensus.cc:3037] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:34.137427 13797 raft_consensus.cc:504] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c57b0b70246b6a39f3b05e4a5e09b" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 38677 } } peers { permanent_uuid: "e43eeb2448b74afeaa80927b9e85ade4" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 43721 } } peers { permanent_uuid: "4b90783690f14d5686e4c02394ca02fd" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37975 } }
I20260926 20:52:34.138111 13797 leader_election.cc:288] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [CANDIDATE]: Term 1 election: Requested vote from peers 305c57b0b70246b6a39f3b05e4a5e09b (127.8.249.130:38677), e43eeb2448b74afeaa80927b9e85ade4 (127.8.249.129:43721)
I20260926 20:52:34.138587 13595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856fc8c732204949bb4cb350f3225be7" candidate_uuid: "4b90783690f14d5686e4c02394ca02fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "305c57b0b70246b6a39f3b05e4a5e09b"
I20260926 20:52:34.138587 13462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856fc8c732204949bb4cb350f3225be7" candidate_uuid: "4b90783690f14d5686e4c02394ca02fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e43eeb2448b74afeaa80927b9e85ade4"
I20260926 20:52:34.138823 13595 raft_consensus.cc:3037] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:34.138827 13462 raft_consensus.cc:3037] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:34.142035 13595 raft_consensus.cc:2454] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b90783690f14d5686e4c02394ca02fd in term 1.
I20260926 20:52:34.142037 13462 raft_consensus.cc:2454] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b90783690f14d5686e4c02394ca02fd in term 1.
I20260926 20:52:34.142484 13663 leader_election.cc:302] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b90783690f14d5686e4c02394ca02fd, e43eeb2448b74afeaa80927b9e85ade4; no voters:
I20260926 20:52:34.142841 13797 raft_consensus.cc:2781] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:34.143314 13797 raft_consensus.cc:686] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [term 1 LEADER]: Becoming Leader. State: Replica: 4b90783690f14d5686e4c02394ca02fd, State: Running, Role: LEADER
I20260926 20:52:34.143828 13797 consensus_queue.cc:237] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c57b0b70246b6a39f3b05e4a5e09b" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 38677 } } peers { permanent_uuid: "e43eeb2448b74afeaa80927b9e85ade4" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 43721 } } peers { permanent_uuid: "4b90783690f14d5686e4c02394ca02fd" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37975 } }
W20260926 20:52:34.145244 13641 tablet.cc:2369] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:52:34.147820 13508 tablet.cc:2369] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:52:34.149025 13318 catalog_manager.cc:5495] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b90783690f14d5686e4c02394ca02fd (127.8.249.131). New cstate: current_term: 1 leader_uuid: "4b90783690f14d5686e4c02394ca02fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "305c57b0b70246b6a39f3b05e4a5e09b" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 38677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e43eeb2448b74afeaa80927b9e85ade4" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 43721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b90783690f14d5686e4c02394ca02fd" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37975 } health_report { overall_health: HEALTHY } } }
W20260926 20:52:34.151731 13773 tablet.cc:2369] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:52:34.198750 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.193071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.8.249.131:37975)
0926 20:52:34.193407 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:34.193413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:34.193518 (+ 105us) server_negotiation.cc:407] Connection header received
0926 20:52:34.193776 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:34.193781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:34.193857 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:34.194016 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:34.194408 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.195205 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.196616 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.197038 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.197171 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:34.197661 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:34.198180 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:34.198343 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:34.198401 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 20:52:34.198482 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260926 20:52:34.207451 13462 raft_consensus.cc:1261] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4 [term 1 FOLLOWER]: Refusing update from remote peer 4b90783690f14d5686e4c02394ca02fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:34.207441 13595 raft_consensus.cc:1261] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b [term 1 FOLLOWER]: Refusing update from remote peer 4b90783690f14d5686e4c02394ca02fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:34.208628 13800 consensus_queue.cc:1035] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e43eeb2448b74afeaa80927b9e85ade4" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 43721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:34.208971 13797 consensus_queue.cc:1035] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "305c57b0b70246b6a39f3b05e4a5e09b" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 38677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:34.226686 13805 mvcc.cc:205] Tried to move back new op lower bound from 7333707588422688768 to 7333707588177416192. Current Snapshot: MvccSnapshot[applied={T|T < 7333707588422688768 or (T in {7333707588422688768})}]
I20260926 20:52:34.235097 13318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:34.235369 13318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:34.239038 13318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 486c9110d3c947468fff9f6601faf50e: Sending DeleteTablet for 3 replicas of tablet 856fc8c732204949bb4cb350f3225be7
I20260926 20:52:34.240211 13442 tablet_service.cc:1514] Processing DeleteTablet for tablet 856fc8c732204949bb4cb350f3225be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:34 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:42064
I20260926 20:52:34.240211 13575 tablet_service.cc:1514] Processing DeleteTablet for tablet 856fc8c732204949bb4cb350f3225be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:34 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:35822
I20260926 20:52:34.241101 13813 tablet_replica.cc:330] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4: stopping tablet replica
I20260926 20:52:34.241590 13813 raft_consensus.cc:2229] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:34.242179 13813 raft_consensus.cc:2258] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:34.242136 13707 tablet_service.cc:1514] Processing DeleteTablet for tablet 856fc8c732204949bb4cb350f3225be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:34 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:50098
I20260926 20:52:34.244606 13813 ts_tablet_manager.cc:1905] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:34.245180 13814 tablet_replica.cc:330] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b: stopping tablet replica
I20260926 20:52:34.245594 13814 raft_consensus.cc:2229] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:34.246061 13814 raft_consensus.cc:2258] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:34.247558 13815 tablet_replica.cc:330] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd: stopping tablet replica
I20260926 20:52:34.247819 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 13380
I20260926 20:52:34.248013 13815 raft_consensus.cc:2229] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:52:34.248667 13815 raft_consensus.cc:2258] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:34.250818 13815 ts_tablet_manager.cc:1905] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 20:52:34 dist-test-slave-2dg3 krb5kdc[13275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:52:34.258455 13815 ts_tablet_manager.cc:1918] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:34.258646 13815 log.cc:1196] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/856fc8c732204949bb4cb350f3225be7
I20260926 20:52:34.263749 13813 ts_tablet_manager.cc:1918] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:34.265472 13814 ts_tablet_manager.cc:1905] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:34.265041 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.249401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.8.249.129:43721)
0926 20:52:34.249559 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:34.249566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:34.249584 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:52:34.249745 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:34.249753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:34.249837 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:34.249914 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:34.251745 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.252464 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.253699 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.254133 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.254263 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:34.258330 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:34.258345 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:34.258360 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:34.258399 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:34.262099 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:34.262782 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:34.262813 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:34.262819 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:34.262986 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:34.263351 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:34.263357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:34.263361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:34.263751 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:34.264057 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:34.264407 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 20:52:34.264660 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:52:34.267616 13815 ts_tablet_manager.cc:1939] T 856fc8c732204949bb4cb350f3225be7 P 4b90783690f14d5686e4c02394ca02fd: Deleting consensus metadata
I20260926 20:52:34.267033 13813 log.cc:1196] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/856fc8c732204949bb4cb350f3225be7
I20260926 20:52:34.267937 13502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:34.270598 13304 catalog_manager.cc:4841] TS 4b90783690f14d5686e4c02394ca02fd (127.8.249.131:37975): tablet 856fc8c732204949bb4cb350f3225be7 (table test-table [id=b9ef86d630cf45b39a152c941221c909]) successfully deleted
I20260926 20:52:34.272850 13814 ts_tablet_manager.cc:1918] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:34.273121 13814 log.cc:1196] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/856fc8c732204949bb4cb350f3225be7
I20260926 20:52:34.273810 13814 ts_tablet_manager.cc:1939] T 856fc8c732204949bb4cb350f3225be7 P 305c57b0b70246b6a39f3b05e4a5e09b: Deleting consensus metadata
I20260926 20:52:34.275852 13303 catalog_manager.cc:4841] TS 305c57b0b70246b6a39f3b05e4a5e09b (127.8.249.130:38677): tablet 856fc8c732204949bb4cb350f3225be7 (table test-table [id=b9ef86d630cf45b39a152c941221c909]) successfully deleted
I20260926 20:52:34.377959 13813 ts_tablet_manager.cc:1939] T 856fc8c732204949bb4cb350f3225be7 P e43eeb2448b74afeaa80927b9e85ade4: Deleting consensus metadata
I20260926 20:52:34.378687 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 13380
W20260926 20:52:34.396471 13306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.249.129:43721 (error 108)
I20260926 20:52:34.396708 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 13513
I20260926 20:52:34.410308 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.409889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:43721 (local address 127.0.0.1:42084)
0926 20:52:34.410065 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:34.410174 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:43721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 20:52:34.410703 13306 catalog_manager.cc:4584] TS e43eeb2448b74afeaa80927b9e85ade4 (127.8.249.129:43721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 856fc8c732204949bb4cb350f3225be7: Network error: Client connection negotiation failed: client connection to 127.8.249.129:43721: connect: Connection refused (error 111)
Sep 26 20:52:34 dist-test-slave-2dg3 krb5kdc[13275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:52:34.418372 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.397808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35826 (local address 127.8.249.130:38677)
0926 20:52:34.397990 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:34.397997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:34.398019 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:52:34.398253 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:34.398262 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:34.398368 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:34.398490 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:34.400813 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.402339 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.405141 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.405979 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.406114 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:34.412480 (+ 6366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:34.412501 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:34.412515 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:34.412552 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:34.415615 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:34.416298 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:34.416304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:34.416307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:34.416432 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:34.416812 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:34.416821 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:34.416823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:34.417210 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:34.417504 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:34.417793 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 20:52:34.418017 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:52:34.419082 13635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:34.464646 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.464081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:43721 (local address 127.0.0.1:42092)
0926 20:52:34.464322 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:34.464464 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:43721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 20:52:34.465853 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.465383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:43721 (local address 127.0.0.1:42098)
0926 20:52:34.465565 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:34.465666 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:43721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 20:52:34.508901 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 13513
I20260926 20:52:34.526279 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.525844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:43721 (local address 127.0.0.1:42108)
0926 20:52:34.526058 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:34.526139 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:43721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 20:52:34.527308 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.526978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:43721 (local address 127.0.0.1:42116)
0926 20:52:34.527123 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:34.527200 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:43721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 20:52:34.531286 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 13645
Sep 26 20:52:34 dist-test-slave-2dg3 krb5kdc[13275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:52:34.547158 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.532138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.8.249.131:37975)
0926 20:52:34.532342 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:34.532351 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:34.532368 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:52:34.532443 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:34.532446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:34.532515 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:34.532626 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:34.533879 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.534690 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.535932 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:34.536384 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:34.536527 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:34.541083 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:34.541096 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:34.541111 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:34.541152 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:34.544353 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:34.545030 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:34.545039 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:34.545041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:34.545210 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:34.545605 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:34.545614 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:34.545624 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:34.545970 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:34.546268 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:34.546572 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 20:52:34.546783 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:52:34.547844 13767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:34.607997 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.607518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:43721 (local address 127.0.0.1:42130)
0926 20:52:34.607746 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:34.607830 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:43721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 20:52:34.608996 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.608630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:43721 (local address 127.0.0.1:42140)
0926 20:52:34.608829 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:34.608893 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:43721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 20:52:34.669263 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 13645
I20260926 20:52:34.688359 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 13288
I20260926 20:52:34.689456 13349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:34.796872 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:34.796157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:43721 (local address 127.0.0.1:42150)
0926 20:52:34.796419 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:34.796553 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:43721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 20:52:34.797118 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 13288
2026-09-26T20:52:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4226 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T20:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:34Z Disabled control of system clock
I20260926 20:52:34.864671 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:40325
--webserver_interface=127.8.249.190
--webserver_port=0
--builtin_ntp_servers=127.8.249.148:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:40325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:52:35.184675 13825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:35.184962 13825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:35.185045 13825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:35.192184 13825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:35.192384 13825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:35.192447 13825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:35.192526 13825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:35.192574 13825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:35.201704 13825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:40325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:40325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:35.203102 13825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:35.204830 13825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:35.217911 13831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:35.217983 13830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:35.218420 13833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:35.218837 13825 server_base.cc:1034] running on GCE node
I20260926 20:52:35.219867 13825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:35.221302 13825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:35.222497 13825 hybrid_clock.cc:648] HybridClock initialized: now 1790455955222470 us; error 30 us; skew 500 ppm
I20260926 20:52:35.222918 13825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:52:35.225170 13825 webserver.cc:460] Webserver started at http://127.8.249.190:39261/ using document root <none> and password file <none>
I20260926 20:52:35.225891 13825 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:35.226014 13825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:35.226346 13825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:35.229252 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "431ed785184b453b89c3c99b9d318665"
format_stamp: "Formatted at 2026-09-26 20:52:35 on dist-test-slave-2dg3"
server_key: "9789becffa2450ad415c1eee27ff87ac"
server_key_iv: "0df2c4e95bcb2cce5913d2ff5d81ce70"
server_key_version: "encryptionkey@0"
I20260926 20:52:35.230053 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "431ed785184b453b89c3c99b9d318665"
format_stamp: "Formatted at 2026-09-26 20:52:35 on dist-test-slave-2dg3"
server_key: "9789becffa2450ad415c1eee27ff87ac"
server_key_iv: "0df2c4e95bcb2cce5913d2ff5d81ce70"
server_key_version: "encryptionkey@0"
I20260926 20:52:35.235056 13825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 20:52:35.238236 13839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:35.239553 13825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:35.239738 13825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "431ed785184b453b89c3c99b9d318665"
format_stamp: "Formatted at 2026-09-26 20:52:35 on dist-test-slave-2dg3"
server_key: "9789becffa2450ad415c1eee27ff87ac"
server_key_iv: "0df2c4e95bcb2cce5913d2ff5d81ce70"
server_key_version: "encryptionkey@0"
I20260926 20:52:35.239985 13825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:35.245478 13825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:35.246248 13825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:35.246554 13825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:35.264412 13825 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:40325
I20260926 20:52:35.264430 13890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:40325 every 8 connection(s)
I20260926 20:52:35.265759 13825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:35.269510 13891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:35.274194 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 13825
I20260926 20:52:35.274389 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:35.275184 9190 external_mini_cluster.cc:1428] Setting key bda394e5d00e7a876b7634c40dd5ad86
I20260926 20:52:35.282107 13891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665: Bootstrap starting.
I20260926 20:52:35.285058 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:35.277356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.8.249.190:40325)
0926 20:52:35.278468 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:35.278483 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:35.278532 (+ 49us) server_negotiation.cc:407] Connection header received
0926 20:52:35.280990 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:35.281017 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:35.281551 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:35.281998 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:35.283060 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:35.283067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:35.283083 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:35.283101 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:35.283168 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:35.283204 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:35.283410 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:35.283531 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 20:52:35.283808 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":349,"threads_started":1}
I20260926 20:52:35.285914 13891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:35.287019 13891 log.cc:824] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:35.289950 13891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665: No bootstrap required, opened a new log
I20260926 20:52:35.296787 13891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431ed785184b453b89c3c99b9d318665" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40325 } }
I20260926 20:52:35.297154 13891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:35.297276 13891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431ed785184b453b89c3c99b9d318665, State: Initialized, Role: FOLLOWER
I20260926 20:52:35.297845 13891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431ed785184b453b89c3c99b9d318665" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40325 } }
I20260926 20:52:35.298080 13891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:35.298205 13891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:35.298368 13891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:35.300390 13891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431ed785184b453b89c3c99b9d318665" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40325 } }
I20260926 20:52:35.300835 13891 leader_election.cc:302] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 431ed785184b453b89c3c99b9d318665; no voters:
I20260926 20:52:35.301383 13891 leader_election.cc:288] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:35.301568 13896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:35.302577 13896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [term 1 LEADER]: Becoming Leader. State: Replica: 431ed785184b453b89c3c99b9d318665, State: Running, Role: LEADER
I20260926 20:52:35.303141 13896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431ed785184b453b89c3c99b9d318665" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40325 } }
I20260926 20:52:35.303503 13891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:35.305907 13897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "431ed785184b453b89c3c99b9d318665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431ed785184b453b89c3c99b9d318665" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40325 } } }
I20260926 20:52:35.306248 13897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:35.306185 13898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 431ed785184b453b89c3c99b9d318665. Latest consensus state: current_term: 1 leader_uuid: "431ed785184b453b89c3c99b9d318665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431ed785184b453b89c3c99b9d318665" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40325 } } }
I20260926 20:52:35.306589 13898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:35.308342 13904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:35.313284 13904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:35.321868 13904 catalog_manager.cc:1347] Generated new cluster ID: 42daec085fe04d8da5d772f838aeb44e
I20260926 20:52:35.322013 13904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:35.330608 13904 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:35.331115 13904 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:35.342938 13904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665: Generated new TSK 0
I20260926 20:52:35.343914 13904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:52:35.357399 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40325
--builtin_ntp_servers=127.8.249.148:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:52:35.709673 13915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:35.709970 13915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:35.710115 13915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:35.717489 13915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:35.717602 13915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:35.717777 13915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:52:35.727080 13915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:35.728381 13915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:35.730144 13915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:35.744163 13920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:35.744163 13923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:35.744134 13921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:35.745172 13915 server_base.cc:1034] running on GCE node
I20260926 20:52:35.745882 13915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:35.747020 13915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:35.748250 13915 hybrid_clock.cc:648] HybridClock initialized: now 1790455955748179 us; error 72 us; skew 500 ppm
I20260926 20:52:35.748588 13915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:52:35.751004 13915 webserver.cc:460] Webserver started at http://127.8.249.129:33089/ using document root <none> and password file <none>
I20260926 20:52:35.751727 13915 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:35.751859 13915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:35.752166 13915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:35.755131 13915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "2afe0d65967d43058109b81259794e59"
format_stamp: "Formatted at 2026-09-26 20:52:35 on dist-test-slave-2dg3"
server_key: "01bf1ed12cc2c94c20b4c6ccb27baa75"
server_key_iv: "d95171f4a29e24ef4e56f32d7a96e31e"
server_key_version: "encryptionkey@0"
I20260926 20:52:35.755927 13915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "2afe0d65967d43058109b81259794e59"
format_stamp: "Formatted at 2026-09-26 20:52:35 on dist-test-slave-2dg3"
server_key: "01bf1ed12cc2c94c20b4c6ccb27baa75"
server_key_iv: "d95171f4a29e24ef4e56f32d7a96e31e"
server_key_version: "encryptionkey@0"
I20260926 20:52:35.761117 13915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:52:35.764619 13929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:35.765959 13915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:52:35.766140 13915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "2afe0d65967d43058109b81259794e59"
format_stamp: "Formatted at 2026-09-26 20:52:35 on dist-test-slave-2dg3"
server_key: "01bf1ed12cc2c94c20b4c6ccb27baa75"
server_key_iv: "d95171f4a29e24ef4e56f32d7a96e31e"
server_key_version: "encryptionkey@0"
I20260926 20:52:35.766384 13915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:35.772428 13915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:35.773294 13915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:35.773588 13915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:35.774585 13915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:35.776155 13915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:35.776283 13915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:35.776413 13915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:35.776504 13915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:35.809825 13915 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:36175
I20260926 20:52:35.809877 14041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:36175 every 8 connection(s)
I20260926 20:52:35.811163 13915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:52:35.819509 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:35.813752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:39529 (local address 127.8.249.190:40325)
0926 20:52:35.814198 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:35.814203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:35.815423 (+ 1220us) server_negotiation.cc:407] Connection header received
0926 20:52:35.817098 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:35.817103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:35.817187 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:35.817308 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:35.818693 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:35.818699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:35.818704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:35.818726 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:35.818774 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:35.818786 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:35.818958 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:35.819146 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 20:52:35.819215 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":247,"threads_started":1}
I20260926 20:52:35.820482 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:35.814266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40325 (local address 127.8.249.129:39529)
0926 20:52:35.815267 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:35.815301 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:35.816818 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:35.817544 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:35.817553 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:35.817646 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:35.818381 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:52:35.818384 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:35.818423 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:52:35.818509 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:35.818917 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:35.818927 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:35.818936 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:52:35.819226 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 20:52:35.819498 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":276,"threads_started":1}
I20260926 20:52:35.820591 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 13915
I20260926 20:52:35.820802 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:52:35.821485 9190 external_mini_cluster.cc:1428] Setting key 2b9534fb06e8e3660a9eece69851805f
I20260926 20:52:35.822101 14042 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40325
I20260926 20:52:35.822441 14042 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:35.823170 14042 heartbeater.cc:511] Master 127.8.249.190:40325 requested a full tablet report, sending...
I20260926 20:52:35.824570 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40325
--builtin_ntp_servers=127.8.249.148:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 20:52:35.825266 13856 ts_manager.cc:194] Registered new tserver with Master: 2afe0d65967d43058109b81259794e59 (127.8.249.129:36175)
W20260926 20:52:36.151345 14046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:36.151672 14046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:36.151849 14046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:36.159230 14046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:36.159382 14046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:36.159576 14046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:52:36.168910 14046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:36.170365 14046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:36.172209 14046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:36.185791 14054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:36.185765 14052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:36.185863 14051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:36.186532 14046 server_base.cc:1034] running on GCE node
I20260926 20:52:36.187197 14046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:36.188153 14046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:36.189348 14046 hybrid_clock.cc:648] HybridClock initialized: now 1790455956189298 us; error 46 us; skew 500 ppm
I20260926 20:52:36.189667 14046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:52:36.191979 14046 webserver.cc:460] Webserver started at http://127.8.249.130:39923/ using document root <none> and password file <none>
I20260926 20:52:36.192651 14046 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:36.192759 14046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:36.193091 14046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:36.195955 14046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "7097919ce7f64f8ab8a69ccae8d2405a"
format_stamp: "Formatted at 2026-09-26 20:52:36 on dist-test-slave-2dg3"
server_key: "8dacb89938609f7cd1f7c2afbf221120"
server_key_iv: "b15119645fab6ecaf4dd34ccc7c35591"
server_key_version: "encryptionkey@0"
I20260926 20:52:36.196727 14046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "7097919ce7f64f8ab8a69ccae8d2405a"
format_stamp: "Formatted at 2026-09-26 20:52:36 on dist-test-slave-2dg3"
server_key: "8dacb89938609f7cd1f7c2afbf221120"
server_key_iv: "b15119645fab6ecaf4dd34ccc7c35591"
server_key_version: "encryptionkey@0"
I20260926 20:52:36.201882 14046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:52:36.205665 14060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:36.207190 14046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:36.207360 14046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "7097919ce7f64f8ab8a69ccae8d2405a"
format_stamp: "Formatted at 2026-09-26 20:52:36 on dist-test-slave-2dg3"
server_key: "8dacb89938609f7cd1f7c2afbf221120"
server_key_iv: "b15119645fab6ecaf4dd34ccc7c35591"
server_key_version: "encryptionkey@0"
I20260926 20:52:36.207623 14046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:36.213554 14046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:36.214357 14046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:36.214620 14046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:36.215637 14046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:36.217033 14046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:36.217118 14046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:36.217267 14046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:36.217356 14046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:36.250166 14046 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:43263
I20260926 20:52:36.250221 14172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:43263 every 8 connection(s)
I20260926 20:52:36.251509 14046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:52:36.254357 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 14046
I20260926 20:52:36.254523 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:52:36.255023 9190 external_mini_cluster.cc:1428] Setting key a78692b3124ab556fbdde88595083b0a
I20260926 20:52:36.257983 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40325
--builtin_ntp_servers=127.8.249.148:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 20:52:36.260602 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.254057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:55033 (local address 127.8.249.190:40325)
0926 20:52:36.254279 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:36.254285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:36.256015 (+ 1730us) server_negotiation.cc:407] Connection header received
0926 20:52:36.258196 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:36.258203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:36.258261 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:36.258387 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:36.259951 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:36.259955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:36.259958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:36.259977 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:36.260010 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:36.260022 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:36.260121 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:36.260347 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 20:52:36.260397 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:52:36.261915 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.254757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40325 (local address 127.8.249.130:55033)
0926 20:52:36.255827 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:36.255893 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:36.257852 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:36.258661 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:36.258676 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:36.258841 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:36.259584 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:52:36.259589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:36.259645 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:52:36.259761 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:36.260125 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:36.260137 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:36.260148 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 20:52:36.260418 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 20:52:36.260686 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":291,"threads_started":1}
I20260926 20:52:36.263299 14173 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40325
I20260926 20:52:36.263610 14173 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:36.264482 14173 heartbeater.cc:511] Master 127.8.249.190:40325 requested a full tablet report, sending...
I20260926 20:52:36.266108 13856 ts_manager.cc:194] Registered new tserver with Master: 7097919ce7f64f8ab8a69ccae8d2405a (127.8.249.130:43263)
W20260926 20:52:36.590196 14176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:36.590523 14176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:36.590724 14176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:36.598052 14176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:36.598196 14176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:36.598387 14176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:52:36.607721 14176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:36.609045 14176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:36.610893 14176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:36.624086 14181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:36.624058 14182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:36.623999 14184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:36.628156 14176 server_base.cc:1034] running on GCE node
I20260926 20:52:36.628696 14176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:36.629563 14176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:36.630852 14176 hybrid_clock.cc:648] HybridClock initialized: now 1790455956630755 us; error 53 us; skew 500 ppm
I20260926 20:52:36.631179 14176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:52:36.633363 14176 webserver.cc:460] Webserver started at http://127.8.249.131:32839/ using document root <none> and password file <none>
I20260926 20:52:36.634047 14176 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:36.634176 14176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:36.634493 14176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:36.637374 14176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "56513da11d7746159e5d7cc4c1e6710e"
format_stamp: "Formatted at 2026-09-26 20:52:36 on dist-test-slave-2dg3"
server_key: "9a5f9cdf1a156fdab98bace067793547"
server_key_iv: "c44b6fa907f55d7a949837d2cd0f85b5"
server_key_version: "encryptionkey@0"
I20260926 20:52:36.638142 14176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "56513da11d7746159e5d7cc4c1e6710e"
format_stamp: "Formatted at 2026-09-26 20:52:36 on dist-test-slave-2dg3"
server_key: "9a5f9cdf1a156fdab98bace067793547"
server_key_iv: "c44b6fa907f55d7a949837d2cd0f85b5"
server_key_version: "encryptionkey@0"
I20260926 20:52:36.643393 14176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 20:52:36.647403 14190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:36.649286 14176 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:52:36.649504 14176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "56513da11d7746159e5d7cc4c1e6710e"
format_stamp: "Formatted at 2026-09-26 20:52:36 on dist-test-slave-2dg3"
server_key: "9a5f9cdf1a156fdab98bace067793547"
server_key_iv: "c44b6fa907f55d7a949837d2cd0f85b5"
server_key_version: "encryptionkey@0"
I20260926 20:52:36.649801 14176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:36.657145 14176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:36.657935 14176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:36.658205 14176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:36.659215 14176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:36.660553 14176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:36.660665 14176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:36.660796 14176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:36.660897 14176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:36.692986 14176 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:34501
I20260926 20:52:36.693034 14302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:34501 every 8 connection(s)
I20260926 20:52:36.694356 14176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:52:36.699946 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 14176
I20260926 20:52:36.700110 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:52:36.700521 9190 external_mini_cluster.cc:1428] Setting key b075b6f5303f45f093a186ca4d531f6d
I20260926 20:52:36.702992 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.697017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:38769 (local address 127.8.249.190:40325)
0926 20:52:36.697214 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:36.697220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:36.698709 (+ 1489us) server_negotiation.cc:407] Connection header received
0926 20:52:36.700555 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:36.700560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:36.700634 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:36.700760 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:36.702174 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:36.702180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:36.702185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:36.702204 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:36.702245 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:36.702256 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:36.702382 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:36.702593 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 20:52:36.702662 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:52:36.704070 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.697656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40325 (local address 127.8.249.131:38769)
0926 20:52:36.698549 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:36.698584 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:36.700268 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:36.701013 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:36.701021 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:36.701119 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:36.701870 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:52:36.701873 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:36.701912 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:52:36.702000 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:36.702365 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:36.702375 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:36.702383 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:52:36.702672 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 20:52:36.702984 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":257,"threads_started":1}
I20260926 20:52:36.705423 14303 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40325
I20260926 20:52:36.705721 14303 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:36.706290 14303 heartbeater.cc:511] Master 127.8.249.190:40325 requested a full tablet report, sending...
I20260926 20:52:36.707773 13856 ts_manager.cc:194] Registered new tserver with Master: 56513da11d7746159e5d7cc4c1e6710e (127.8.249.131:34501)
I20260926 20:52:36.715773 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:52:36.722508 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.720186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58706 (local address 127.8.249.190:40325)
0926 20:52:36.720414 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:36.720422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:36.720806 (+ 384us) server_negotiation.cc:407] Connection header received
0926 20:52:36.720967 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:36.720971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:36.721022 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:36.721102 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:36.721941 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:36.721945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:36.721949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:36.721965 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:36.721993 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:36.722004 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:36.722096 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:36.722160 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 20:52:36.722202 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:52:36.726300 13856 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:52:36.732959 13856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:52:36.750207 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.745131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.8.249.131:34501)
0926 20:52:36.745945 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:36.745958 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:36.746407 (+ 449us) server_negotiation.cc:407] Connection header received
0926 20:52:36.746669 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:36.746678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:36.746942 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:36.747064 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:36.749273 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:36.749281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:36.749294 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:36.749312 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:36.749371 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:36.749393 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:36.749552 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:36.749796 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 20:52:36.749895 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":407,"threads_started":1}
I20260926 20:52:36.750355 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.745415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:43263 (local address 127.0.0.1:33310)
0926 20:52:36.746412 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:36.746432 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:36.746535 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:36.747920 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:36.747929 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:36.748016 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:36.748905 (+ 889us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:52:36.748914 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:36.748945 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:52:36.749044 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:36.749537 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:36.749548 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:36.749557 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:52:36.749717 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 20:52:36.749811 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":797,"threads_started":1}
I20260926 20:52:36.751031 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.744878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:34501 (local address 127.0.0.1:39228)
0926 20:52:36.746272 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:36.746316 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:36.746517 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:36.748285 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:36.748290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:36.748314 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:36.749040 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:52:36.749044 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:36.749061 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:52:36.749109 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:36.749640 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:36.749644 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:36.749648 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:52:36.750526 (+ 878us) client_negotiation.cc:240] Negotiation successful
0926 20:52:36.750585 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"thread_start_us":1242,"threads_started":1}
I20260926 20:52:36.751559 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.745640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.8.249.130:43263)
0926 20:52:36.747186 (+ 1546us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:36.747192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:36.747217 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:52:36.747281 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:36.747286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:36.747484 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:36.747660 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:36.749261 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:36.749272 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:36.749289 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:36.749318 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:36.749380 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:36.749403 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:36.750970 (+ 1567us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:36.751048 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 20:52:36.751146 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1394,"thread_start_us":1270,"threads_started":1}
I20260926 20:52:36.753579 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.747223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.8.249.129:36175)
0926 20:52:36.747783 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:36.747792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:36.749051 (+ 1259us) server_negotiation.cc:407] Connection header received
0926 20:52:36.749314 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:36.749323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:36.749517 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:36.749633 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:36.752672 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:36.752680 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:36.752695 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:36.752723 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:36.752783 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:36.752805 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:36.752930 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:36.753142 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 20:52:36.753230 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":395,"threads_started":1}
I20260926 20:52:36.754276 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.747490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:36175 (local address 127.0.0.1:56456)
0926 20:52:36.748927 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:36.748944 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:36.749140 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:36.750596 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:36.750601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:36.750629 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:36.751570 (+ 941us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:52:36.751575 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:36.751596 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:52:36.751642 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:36.752984 (+ 1342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:36.752988 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:36.752993 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:52:36.753801 (+ 808us) client_negotiation.cc:240] Negotiation successful
0926 20:52:36.753849 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1308,"thread_start_us":2235,"threads_started":1}
I20260926 20:52:36.755270 14108 tablet_service.cc:1467] Processing CreateTablet for tablet 7622f2d2fc494747b23e9be9d71edf5b (DEFAULT_TABLE table=test-table [id=08dd6b33351e42c9915cbae780968273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:36.755190 14238 tablet_service.cc:1467] Processing CreateTablet for tablet 7622f2d2fc494747b23e9be9d71edf5b (DEFAULT_TABLE table=test-table [id=08dd6b33351e42c9915cbae780968273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:36.756587 14238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7622f2d2fc494747b23e9be9d71edf5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:36.756630 14108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7622f2d2fc494747b23e9be9d71edf5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:36.758292 13977 tablet_service.cc:1467] Processing CreateTablet for tablet 7622f2d2fc494747b23e9be9d71edf5b (DEFAULT_TABLE table=test-table [id=08dd6b33351e42c9915cbae780968273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:36.759625 13977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7622f2d2fc494747b23e9be9d71edf5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:36.769646 14321 tablet_bootstrap.cc:492] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e: Bootstrap starting.
I20260926 20:52:36.770701 14322 tablet_bootstrap.cc:492] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a: Bootstrap starting.
I20260926 20:52:36.773447 14323 tablet_bootstrap.cc:492] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59: Bootstrap starting.
I20260926 20:52:36.773901 14321 tablet_bootstrap.cc:654] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:36.774497 14322 tablet_bootstrap.cc:654] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:36.775802 14322 log.cc:824] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:36.775292 14321 log.cc:824] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:36.776862 14323 tablet_bootstrap.cc:654] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:36.778059 14323 log.cc:824] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:36.779827 14321 tablet_bootstrap.cc:492] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e: No bootstrap required, opened a new log
I20260926 20:52:36.779896 14322 tablet_bootstrap.cc:492] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a: No bootstrap required, opened a new log
I20260926 20:52:36.780143 14321 ts_tablet_manager.cc:1397] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260926 20:52:36.780186 14322 ts_tablet_manager.cc:1397] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 20:52:36.780854 14323 tablet_bootstrap.cc:492] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59: No bootstrap required, opened a new log
I20260926 20:52:36.781095 14323 ts_tablet_manager.cc:1397] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260926 20:52:36.787920 14321 raft_consensus.cc:348] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56513da11d7746159e5d7cc4c1e6710e" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 34501 } } peers { permanent_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 43263 } } peers { permanent_uuid: "2afe0d65967d43058109b81259794e59" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36175 } }
I20260926 20:52:36.788304 14321 raft_consensus.cc:374] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:36.788442 14321 raft_consensus.cc:729] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56513da11d7746159e5d7cc4c1e6710e, State: Initialized, Role: FOLLOWER
I20260926 20:52:36.789067 14321 consensus_queue.cc:260] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56513da11d7746159e5d7cc4c1e6710e" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 34501 } } peers { permanent_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 43263 } } peers { permanent_uuid: "2afe0d65967d43058109b81259794e59" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36175 } }
I20260926 20:52:36.790221 14303 heartbeater.cc:503] Master 127.8.249.190:40325 was elected leader, sending a full tablet report...
I20260926 20:52:36.790752 14321 ts_tablet_manager.cc:1428] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260926 20:52:36.790508 14322 raft_consensus.cc:348] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56513da11d7746159e5d7cc4c1e6710e" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 34501 } } peers { permanent_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 43263 } } peers { permanent_uuid: "2afe0d65967d43058109b81259794e59" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36175 } }
I20260926 20:52:36.791064 14322 raft_consensus.cc:374] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:36.791150 14322 raft_consensus.cc:729] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7097919ce7f64f8ab8a69ccae8d2405a, State: Initialized, Role: FOLLOWER
I20260926 20:52:36.791721 14322 consensus_queue.cc:260] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56513da11d7746159e5d7cc4c1e6710e" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 34501 } } peers { permanent_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 43263 } } peers { permanent_uuid: "2afe0d65967d43058109b81259794e59" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36175 } }
I20260926 20:52:36.792243 14323 raft_consensus.cc:348] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56513da11d7746159e5d7cc4c1e6710e" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 34501 } } peers { permanent_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 43263 } } peers { permanent_uuid: "2afe0d65967d43058109b81259794e59" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36175 } }
I20260926 20:52:36.792690 14323 raft_consensus.cc:374] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:36.792814 14323 raft_consensus.cc:729] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2afe0d65967d43058109b81259794e59, State: Initialized, Role: FOLLOWER
I20260926 20:52:36.792876 14173 heartbeater.cc:503] Master 127.8.249.190:40325 was elected leader, sending a full tablet report...
I20260926 20:52:36.793403 14322 ts_tablet_manager.cc:1428] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260926 20:52:36.793382 14323 consensus_queue.cc:260] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56513da11d7746159e5d7cc4c1e6710e" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 34501 } } peers { permanent_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 43263 } } peers { permanent_uuid: "2afe0d65967d43058109b81259794e59" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36175 } }
I20260926 20:52:36.794461 14042 heartbeater.cc:503] Master 127.8.249.190:40325 was elected leader, sending a full tablet report...
I20260926 20:52:36.795665 14323 ts_tablet_manager.cc:1428] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
W20260926 20:52:36.813364 14043 tablet.cc:2369] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:52:36.877799 14328 raft_consensus.cc:482] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:36.878180 14328 raft_consensus.cc:504] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56513da11d7746159e5d7cc4c1e6710e" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 34501 } } peers { permanent_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 43263 } } peers { permanent_uuid: "2afe0d65967d43058109b81259794e59" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36175 } }
I20260926 20:52:36.880280 14328 leader_election.cc:288] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56513da11d7746159e5d7cc4c1e6710e (127.8.249.131:34501), 2afe0d65967d43058109b81259794e59 (127.8.249.129:36175)
I20260926 20:52:36.883813 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.880564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:33899 (local address 127.8.249.131:34501)
0926 20:52:36.880782 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:36.880792 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:36.881608 (+ 816us) server_negotiation.cc:407] Connection header received
0926 20:52:36.881793 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:36.881800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:36.881858 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:36.881930 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:36.883154 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:36.883161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:36.883167 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:36.883188 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:36.883225 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:36.883240 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:36.883345 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:36.883488 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 20:52:36.883551 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:52:36.883975 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.880803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:36339 (local address 127.8.249.129:36175)
0926 20:52:36.881013 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:36.881022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:36.881686 (+ 664us) server_negotiation.cc:407] Connection header received
0926 20:52:36.881839 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:36.881843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:36.881895 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:36.882026 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:36.883264 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:36.883268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:36.883271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:36.883306 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:36.883349 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:36.883366 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:36.883492 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:36.883617 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 20:52:36.883675 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:52:36.884124 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.880468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:34501 (local address 127.8.249.130:33899)
0926 20:52:36.881459 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:36.881483 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:36.881646 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:36.881997 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:36.882005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:36.882029 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:36.882949 (+ 920us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:52:36.882952 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:36.882966 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:52:36.883008 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:36.883347 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:36.883352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:36.883358 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:52:36.883536 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 20:52:36.883614 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":533,"threads_started":1}
I20260926 20:52:36.884555 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.880798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:36175 (local address 127.8.249.130:36339)
0926 20:52:36.881533 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:36.881548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:36.881685 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:36.882072 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:36.882077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:36.882107 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:36.883031 (+ 924us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:52:36.883033 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:36.883045 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:52:36.883106 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:36.883483 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:36.883487 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:36.883491 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:52:36.883631 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 20:52:36.883680 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":457,"threads_started":1}
I20260926 20:52:36.885213 14258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7622f2d2fc494747b23e9be9d71edf5b" candidate_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56513da11d7746159e5d7cc4c1e6710e" is_pre_election: true
I20260926 20:52:36.885352 13997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7622f2d2fc494747b23e9be9d71edf5b" candidate_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afe0d65967d43058109b81259794e59" is_pre_election: true
I20260926 20:52:36.885744 14258 raft_consensus.cc:2454] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7097919ce7f64f8ab8a69ccae8d2405a in term 0.
I20260926 20:52:36.885850 13997 raft_consensus.cc:2454] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7097919ce7f64f8ab8a69ccae8d2405a in term 0.
I20260926 20:52:36.886478 14064 leader_election.cc:302] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2afe0d65967d43058109b81259794e59, 56513da11d7746159e5d7cc4c1e6710e, 7097919ce7f64f8ab8a69ccae8d2405a; no voters:
I20260926 20:52:36.886888 14328 raft_consensus.cc:2781] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:52:36.887032 14328 raft_consensus.cc:482] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:36.887152 14328 raft_consensus.cc:3037] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:36.889329 14328 raft_consensus.cc:504] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56513da11d7746159e5d7cc4c1e6710e" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 34501 } } peers { permanent_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 43263 } } peers { permanent_uuid: "2afe0d65967d43058109b81259794e59" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36175 } }
I20260926 20:52:36.889909 14328 leader_election.cc:288] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [CANDIDATE]: Term 1 election: Requested vote from peers 56513da11d7746159e5d7cc4c1e6710e (127.8.249.131:34501), 2afe0d65967d43058109b81259794e59 (127.8.249.129:36175)
I20260926 20:52:36.890496 14258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7622f2d2fc494747b23e9be9d71edf5b" candidate_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56513da11d7746159e5d7cc4c1e6710e"
I20260926 20:52:36.890487 13997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7622f2d2fc494747b23e9be9d71edf5b" candidate_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afe0d65967d43058109b81259794e59"
I20260926 20:52:36.890695 13997 raft_consensus.cc:3037] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:36.890695 14258 raft_consensus.cc:3037] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:36.893888 14258 raft_consensus.cc:2454] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7097919ce7f64f8ab8a69ccae8d2405a in term 1.
I20260926 20:52:36.893890 13997 raft_consensus.cc:2454] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7097919ce7f64f8ab8a69ccae8d2405a in term 1.
I20260926 20:52:36.894279 14064 leader_election.cc:302] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56513da11d7746159e5d7cc4c1e6710e, 7097919ce7f64f8ab8a69ccae8d2405a; no voters:
I20260926 20:52:36.894591 14328 raft_consensus.cc:2781] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:36.895157 14328 raft_consensus.cc:686] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [term 1 LEADER]: Becoming Leader. State: Replica: 7097919ce7f64f8ab8a69ccae8d2405a, State: Running, Role: LEADER
I20260926 20:52:36.895709 14328 consensus_queue.cc:237] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56513da11d7746159e5d7cc4c1e6710e" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 34501 } } peers { permanent_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 43263 } } peers { permanent_uuid: "2afe0d65967d43058109b81259794e59" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36175 } }
I20260926 20:52:36.900701 13854 catalog_manager.cc:5495] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7097919ce7f64f8ab8a69ccae8d2405a (127.8.249.130). New cstate: current_term: 1 leader_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56513da11d7746159e5d7cc4c1e6710e" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 34501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7097919ce7f64f8ab8a69ccae8d2405a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 43263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2afe0d65967d43058109b81259794e59" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36175 } health_report { overall_health: UNKNOWN } } }
I20260926 20:52:36.944680 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.942400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.8.249.130:43263)
0926 20:52:36.942584 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:36.942590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:36.942709 (+ 119us) server_negotiation.cc:407] Connection header received
0926 20:52:36.943017 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:36.943021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:36.943099 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:36.943215 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:36.944091 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:36.944101 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:36.944104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:36.944119 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:36.944147 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:36.944158 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:36.944288 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:36.944368 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:52:36.944426 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 20:52:36.947219 14304 tablet.cc:2369] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:52:36.952406 14258 raft_consensus.cc:1261] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e [term 1 FOLLOWER]: Refusing update from remote peer 7097919ce7f64f8ab8a69ccae8d2405a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:36.952405 13997 raft_consensus.cc:1261] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59 [term 1 FOLLOWER]: Refusing update from remote peer 7097919ce7f64f8ab8a69ccae8d2405a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:36.953346 14332 consensus_queue.cc:1035] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [LEADER]: Connected to new peer: Peer: permanent_uuid: "56513da11d7746159e5d7cc4c1e6710e" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 34501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:36.953645 14328 consensus_queue.cc:1035] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2afe0d65967d43058109b81259794e59" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:36.984258 13855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:36.984581 13855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:36.988410 13855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 431ed785184b453b89c3c99b9d318665: Sending DeleteTablet for 3 replicas of tablet 7622f2d2fc494747b23e9be9d71edf5b
I20260926 20:52:36.989494 14238 tablet_service.cc:1514] Processing DeleteTablet for tablet 7622f2d2fc494747b23e9be9d71edf5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:36 UTC) from {username='slave'} at 127.0.0.1:39228
I20260926 20:52:36.989594 13977 tablet_service.cc:1514] Processing DeleteTablet for tablet 7622f2d2fc494747b23e9be9d71edf5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:36 UTC) from {username='slave'} at 127.0.0.1:56456
I20260926 20:52:36.989732 14108 tablet_service.cc:1514] Processing DeleteTablet for tablet 7622f2d2fc494747b23e9be9d71edf5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:36 UTC) from {username='slave'} at 127.0.0.1:33310
I20260926 20:52:36.991982 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 13915
I20260926 20:52:37.005045 14345 tablet_replica.cc:330] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e: stopping tablet replica
I20260926 20:52:37.005560 14345 raft_consensus.cc:2229] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:37.006109 14345 raft_consensus.cc:2258] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:37.008555 14345 ts_tablet_manager.cc:1905] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:37.015492 14345 ts_tablet_manager.cc:1918] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:37.015666 14345 log.cc:1196] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/7622f2d2fc494747b23e9be9d71edf5b
I20260926 20:52:37.016176 14345 ts_tablet_manager.cc:1939] T 7622f2d2fc494747b23e9be9d71edf5b P 56513da11d7746159e5d7cc4c1e6710e: Deleting consensus metadata
I20260926 20:52:37.018081 13843 catalog_manager.cc:4841] TS 56513da11d7746159e5d7cc4c1e6710e (127.8.249.131:34501): tablet 7622f2d2fc494747b23e9be9d71edf5b (table test-table [id=08dd6b33351e42c9915cbae780968273]) successfully deleted
I20260926 20:52:37.020572 14346 tablet_replica.cc:330] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59: stopping tablet replica
I20260926 20:52:37.021021 14346 raft_consensus.cc:2229] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:37.021525 14346 raft_consensus.cc:2258] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:37.022136 14347 tablet_replica.cc:330] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a: stopping tablet replica
I20260926 20:52:37.022548 14347 raft_consensus.cc:2229] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:52:37.023200 14347 raft_consensus.cc:2258] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:37.023486 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:36.993724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56468 (local address 127.8.249.129:36175)
0926 20:52:36.993957 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:36.993967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:37.018198 (+ 24231us) server_negotiation.cc:407] Connection header received
0926 20:52:37.020429 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:37.020435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:37.020612 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:37.020854 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:37.022614 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:37.022619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:37.022624 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:37.022642 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:37.022681 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:37.022696 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:37.022821 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:37.023133 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 20:52:37.023191 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:52:37.024631 14037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:37.025153 14347 ts_tablet_manager.cc:1905] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:37.034533 14347 ts_tablet_manager.cc:1918] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:37.034762 14347 log.cc:1196] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/7622f2d2fc494747b23e9be9d71edf5b
I20260926 20:52:37.035315 14347 ts_tablet_manager.cc:1939] T 7622f2d2fc494747b23e9be9d71edf5b P 7097919ce7f64f8ab8a69ccae8d2405a: Deleting consensus metadata
I20260926 20:52:37.036924 13841 catalog_manager.cc:4841] TS 7097919ce7f64f8ab8a69ccae8d2405a (127.8.249.130:43263): tablet 7622f2d2fc494747b23e9be9d71edf5b (table test-table [id=08dd6b33351e42c9915cbae780968273]) successfully deleted
I20260926 20:52:37.024894 14346 ts_tablet_manager.cc:1905] T 7622f2d2fc494747b23e9be9d71edf5b P 2afe0d65967d43058109b81259794e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:37.141566 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 13915
I20260926 20:52:37.158368 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 14046
W20260926 20:52:37.158481 13841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.249.129:36175 (error 108)
I20260926 20:52:37.160531 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:37.159979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:36175 (local address 127.0.0.1:56470)
0926 20:52:37.160211 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:37.160336 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260926 20:52:37.160774 13841 catalog_manager.cc:4584] TS 2afe0d65967d43058109b81259794e59 (127.8.249.129:36175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7622f2d2fc494747b23e9be9d71edf5b: Network error: Client connection negotiation failed: client connection to 127.8.249.129:36175: connect: Connection refused (error 111)
I20260926 20:52:37.161710 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:37.159527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.8.249.130:43263)
0926 20:52:37.159709 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:37.159720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:37.159741 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:52:37.159823 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:37.159827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:37.159876 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:37.160000 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:37.161090 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:37.161094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:37.161098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:37.161113 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:37.161141 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:37.161152 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:37.161295 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:37.161387 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 20:52:37.161448 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:52:37.162425 14168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:37.218731 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:37.218107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:36175 (local address 127.0.0.1:56472)
0926 20:52:37.218361 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:37.218497 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 20:52:37.219931 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:37.219541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:36175 (local address 127.0.0.1:56478)
0926 20:52:37.219690 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:37.219794 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 20:52:37.279008 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 14046
I20260926 20:52:37.281385 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:37.280858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:36175 (local address 127.0.0.1:56492)
0926 20:52:37.281078 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:37.281212 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 20:52:37.282483 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:37.282042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:36175 (local address 127.0.0.1:56498)
0926 20:52:37.282217 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:37.282310 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:52:37.297253 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 14176
I20260926 20:52:37.299957 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:37.297972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39240 (local address 127.8.249.131:34501)
0926 20:52:37.298143 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:37.298148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:37.298165 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:52:37.298315 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:37.298319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:37.298401 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:37.298490 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:37.299370 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:37.299374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:37.299376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:37.299388 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:37.299413 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:37.299424 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:37.299530 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:37.299609 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 20:52:37.299651 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:52:37.300591 14298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:37.363641 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:37.363129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:36175 (local address 127.0.0.1:56508)
0926 20:52:37.363365 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:37.363475 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 20:52:37.364652 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:37.364310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:36175 (local address 127.0.0.1:56510)
0926 20:52:37.364486 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:37.364548 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 20:52:37.387459 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 14176
I20260926 20:52:37.403209 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 13825
I20260926 20:52:37.404337 13886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:37.508507 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 13825
2026-09-26T20:52:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2711 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 20:52:37.550189 9190 mini_oidc.cc:152] Starting JWKS server
I20260926 20:52:37.553188 9190 webserver.cc:460] Webserver started at https://127.0.0.1:40145/ using document root <none> and password file <none>
I20260926 20:52:37.553577 9190 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:52:37.554333 9190 webserver.cc:460] Webserver started at http://127.0.0.1:33581/ using document root <none> and password file <none>
2026-09-26T20:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:37Z Disabled control of system clock
I20260926 20:52:37.584360 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:41867
--webserver_interface=127.8.249.190
--webserver_port=0
--builtin_ntp_servers=127.8.249.148:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:41867
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40145/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/testchainca.cert with env {}
W20260926 20:52:37.909176 14359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:37.909487 14359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:37.909574 14359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:37.916989 14359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:37.917109 14359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:37.917146 14359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:37.917205 14359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:37.917249 14359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:52:37.917303 14359 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:52:37.917342 14359 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40145/jwks.json
I20260926 20:52:37.927386 14359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:41867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:41867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40145/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:37.928656 14359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:37.929886 14359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:37.941931 14367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:37.941903 14365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:37.942085 14364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:37.942880 14359 server_base.cc:1034] running on GCE node
I20260926 20:52:37.943753 14359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:37.945246 14359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:37.946460 14359 hybrid_clock.cc:648] HybridClock initialized: now 1790455957946419 us; error 33 us; skew 500 ppm
I20260926 20:52:37.946925 14359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:52:37.949327 14359 webserver.cc:460] Webserver started at http://127.8.249.190:39509/ using document root <none> and password file <none>
I20260926 20:52:37.950107 14359 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:37.950234 14359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:37.950578 14359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:37.953584 14359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "08a66d97d87040df82db53def3e950c9"
format_stamp: "Formatted at 2026-09-26 20:52:37 on dist-test-slave-2dg3"
server_key: "e9654a8f7ee07bd7986456ca37f047a9"
server_key_iv: "436d3bcb8569355b50c77d816719a36d"
server_key_version: "encryptionkey@0"
I20260926 20:52:37.954360 14359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "08a66d97d87040df82db53def3e950c9"
format_stamp: "Formatted at 2026-09-26 20:52:37 on dist-test-slave-2dg3"
server_key: "e9654a8f7ee07bd7986456ca37f047a9"
server_key_iv: "436d3bcb8569355b50c77d816719a36d"
server_key_version: "encryptionkey@0"
I20260926 20:52:37.959415 14359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:52:37.962864 14373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:37.964139 14359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 20:52:37.964330 14359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "08a66d97d87040df82db53def3e950c9"
format_stamp: "Formatted at 2026-09-26 20:52:37 on dist-test-slave-2dg3"
server_key: "e9654a8f7ee07bd7986456ca37f047a9"
server_key_iv: "436d3bcb8569355b50c77d816719a36d"
server_key_version: "encryptionkey@0"
I20260926 20:52:37.964541 14359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:37.977304 14359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:37.978087 14359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:37.978385 14359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:37.997058 14359 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:41867
I20260926 20:52:37.997082 14424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:41867 every 8 connection(s)
I20260926 20:52:37.998463 14359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:38.002377 14425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:38.003229 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 14359
I20260926 20:52:38.003470 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:38.004076 9190 external_mini_cluster.cc:1428] Setting key c34f60a554ca51fdb24e7ce01dda6d83
I20260926 20:52:38.014609 14425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9: Bootstrap starting.
I20260926 20:52:38.017171 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:38.005908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.8.249.190:41867)
0926 20:52:38.006971 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:38.006981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:38.007023 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:52:38.008624 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:38.008645 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:38.009223 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:38.009612 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:38.010167 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:38.011468 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:38.012807 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:38.013444 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:38.013646 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:38.014912 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:38.014924 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:38.014938 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:38.015457 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:38.015536 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:38.015555 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:38.015739 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:38.015971 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 20:52:38.016272 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":363,"threads_started":1}
I20260926 20:52:38.018057 14425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:38.019194 14425 log.cc:824] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:38.021075 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:52:38.022107 14425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9: No bootstrap required, opened a new log
I20260926 20:52:38.029690 14425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a66d97d87040df82db53def3e950c9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41867 } }
I20260926 20:52:38.030063 14425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:38.030159 14425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a66d97d87040df82db53def3e950c9, State: Initialized, Role: FOLLOWER
I20260926 20:52:38.030864 14425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a66d97d87040df82db53def3e950c9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41867 } }
I20260926 20:52:38.031100 14425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:38.031229 14425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:38.031414 14425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:38.033486 14425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a66d97d87040df82db53def3e950c9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41867 } }
I20260926 20:52:38.033982 14425 leader_election.cc:302] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08a66d97d87040df82db53def3e950c9; no voters:
I20260926 20:52:38.034631 14425 leader_election.cc:288] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:38.034766 14430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:38.035561 14430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [term 1 LEADER]: Becoming Leader. State: Replica: 08a66d97d87040df82db53def3e950c9, State: Running, Role: LEADER
I20260926 20:52:38.036063 14430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a66d97d87040df82db53def3e950c9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41867 } }
I20260926 20:52:38.036710 14425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:38.039152 14432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08a66d97d87040df82db53def3e950c9. Latest consensus state: current_term: 1 leader_uuid: "08a66d97d87040df82db53def3e950c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a66d97d87040df82db53def3e950c9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41867 } } }
I20260926 20:52:38.039767 14432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:38.039330 14431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08a66d97d87040df82db53def3e950c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08a66d97d87040df82db53def3e950c9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41867 } } }
I20260926 20:52:38.040936 14431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:38.041684 14437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:38.047259 14437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:38.056993 14437 catalog_manager.cc:1347] Generated new cluster ID: 099834a8931c420fafa267e12300bc8c
I20260926 20:52:38.057114 14437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:38.081084 14437 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:38.082573 14437 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:38.094851 14437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08a66d97d87040df82db53def3e950c9: Generated new TSK 0
I20260926 20:52:38.095690 14437 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:52:38.136428 14455 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 20:52:38.136603 14455 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:52:38.137709 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:38.117231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.8.249.190:41867)
0926 20:52:38.117446 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:38.117454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:38.117955 (+ 501us) server_negotiation.cc:407] Connection header received
0926 20:52:38.118311 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:38.118316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:38.118396 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:38.118553 (+ 157us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:52:38.119157 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:38.119914 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:38.121349 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:38.121816 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:38.121970 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:38.122504 (+ 534us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:52:38.137021 (+ 14517us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:52:38.137386 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 20:52:38.137835 14428 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 20:52:38.139232 14454 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:52:38.117111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:41867 (local address 127.0.0.1:50374)
0926 20:52:38.117837 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:38.117857 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:38.118058 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:38.118573 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:38.118578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:38.118590 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:52:38.119001 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:38.119011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:38.120051 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:38.120059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:38.121197 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:38.121207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:38.122007 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:38.122013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:38.122157 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:38.122221 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:52:38.137282 (+ 15061us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:52:38.137642 (+ 360us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.249.190:41867: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":326,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.249.190:41867: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40145/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 20:52:38.140882 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 14359
I20260926 20:52:38.142064 14420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:38.237185 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 14359
2026-09-26T20:52:38Z chronyd exiting
I20260926 20:52:38.284113 9190 test_util.cc:175] -----------------------------------------------
I20260926 20:52:38.284250 9190 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790455916331309-9190-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (738 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 20:52:38.287853 9190 mini_oidc.cc:152] Starting JWKS server
I20260926 20:52:38.289600 9190 webserver.cc:460] Webserver started at https://127.0.0.1:34491/ using document root <none> and password file <none>
I20260926 20:52:38.289793 9190 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:52:38.290427 9190 webserver.cc:460] Webserver started at http://127.0.0.1:39139/ using document root <none> and password file <none>
2026-09-26T20:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:38Z Disabled control of system clock
I20260926 20:52:38.317021 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:34511
--webserver_interface=127.8.249.190
--webserver_port=0
--builtin_ntp_servers=127.8.249.148:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:34511
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34491/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/testchainca.cert with env {}
W20260926 20:52:38.633638 14462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:38.633910 14462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:38.633996 14462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:38.641037 14462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:38.641140 14462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:38.641181 14462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:38.641230 14462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:38.641265 14462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:52:38.641326 14462 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:52:38.641373 14462 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34491/jwks.json
I20260926 20:52:38.651229 14462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:34511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:34511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34491/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:38.652412 14462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:38.653616 14462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:38.665593 14468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:38.665691 14470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:38.665670 14467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:38.668187 14462 server_base.cc:1034] running on GCE node
I20260926 20:52:38.668941 14462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:38.670295 14462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:38.671506 14462 hybrid_clock.cc:648] HybridClock initialized: now 1790455958671462 us; error 64 us; skew 500 ppm
I20260926 20:52:38.671835 14462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:52:38.677083 14462 webserver.cc:460] Webserver started at http://127.8.249.190:39229/ using document root <none> and password file <none>
I20260926 20:52:38.677806 14462 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:38.677939 14462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:38.678270 14462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:38.681213 14462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "78e7ef1b0d1441cdbd9a8434ad2e67f9"
format_stamp: "Formatted at 2026-09-26 20:52:38 on dist-test-slave-2dg3"
server_key: "2daa862172361e24018ff5535bd968e0"
server_key_iv: "8d9f9c869923648afe95460917c90f30"
server_key_version: "encryptionkey@0"
I20260926 20:52:38.681998 14462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "78e7ef1b0d1441cdbd9a8434ad2e67f9"
format_stamp: "Formatted at 2026-09-26 20:52:38 on dist-test-slave-2dg3"
server_key: "2daa862172361e24018ff5535bd968e0"
server_key_iv: "8d9f9c869923648afe95460917c90f30"
server_key_version: "encryptionkey@0"
I20260926 20:52:38.687223 14462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:52:38.690584 14476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:38.692160 14462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:52:38.692293 14462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "78e7ef1b0d1441cdbd9a8434ad2e67f9"
format_stamp: "Formatted at 2026-09-26 20:52:38 on dist-test-slave-2dg3"
server_key: "2daa862172361e24018ff5535bd968e0"
server_key_iv: "8d9f9c869923648afe95460917c90f30"
server_key_version: "encryptionkey@0"
I20260926 20:52:38.692468 14462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:38.710502 14462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:38.711378 14462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:38.711642 14462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:38.729589 14462 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:34511
I20260926 20:52:38.729609 14527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:34511 every 8 connection(s)
I20260926 20:52:38.730863 14462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:38.734603 14528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:38.735118 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 14462
I20260926 20:52:38.735313 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:38.735917 9190 external_mini_cluster.cc:1428] Setting key 0780ac0b581c340e2ba5df7971f342ca
I20260926 20:52:38.747311 14528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9: Bootstrap starting.
I20260926 20:52:38.750679 14531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:38.737754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.8.249.190:34511)
0926 20:52:38.738770 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:38.738780 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:38.738874 (+ 94us) server_negotiation.cc:407] Connection header received
0926 20:52:38.740891 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:38.740911 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:38.741403 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:38.741762 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:38.742255 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:38.743478 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:38.744721 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:38.745379 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:38.745553 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:38.746856 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:38.746867 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:38.746901 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:38.747688 (+ 787us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:38.747804 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:38.747828 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:38.747961 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:38.748601 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 20:52:38.749070 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":393,"threads_started":1}
I20260926 20:52:38.752030 14528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:38.753490 14528 log.cc:824] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:38.756942 14528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9: No bootstrap required, opened a new log
I20260926 20:52:38.757220 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:52:38.767091 14528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e7ef1b0d1441cdbd9a8434ad2e67f9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 34511 } }
I20260926 20:52:38.767629 14528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:38.767771 14528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e7ef1b0d1441cdbd9a8434ad2e67f9, State: Initialized, Role: FOLLOWER
I20260926 20:52:38.768496 14528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e7ef1b0d1441cdbd9a8434ad2e67f9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 34511 } }
I20260926 20:52:38.768822 14528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:38.769001 14528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:38.769181 14528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:38.772037 14528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e7ef1b0d1441cdbd9a8434ad2e67f9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 34511 } }
I20260926 20:52:38.772688 14528 leader_election.cc:302] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78e7ef1b0d1441cdbd9a8434ad2e67f9; no voters:
I20260926 20:52:38.773445 14528 leader_election.cc:288] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:38.773588 14533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:38.774322 14533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [term 1 LEADER]: Becoming Leader. State: Replica: 78e7ef1b0d1441cdbd9a8434ad2e67f9, State: Running, Role: LEADER
I20260926 20:52:38.774995 14533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e7ef1b0d1441cdbd9a8434ad2e67f9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 34511 } }
I20260926 20:52:38.776077 14528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:38.778185 14535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78e7ef1b0d1441cdbd9a8434ad2e67f9. Latest consensus state: current_term: 1 leader_uuid: "78e7ef1b0d1441cdbd9a8434ad2e67f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e7ef1b0d1441cdbd9a8434ad2e67f9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 34511 } } }
I20260926 20:52:38.778752 14535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:38.778669 14534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78e7ef1b0d1441cdbd9a8434ad2e67f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e7ef1b0d1441cdbd9a8434ad2e67f9" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 34511 } } }
I20260926 20:52:38.779433 14534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:38.782856 14540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:38.790521 14540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:38.803632 14540 catalog_manager.cc:1347] Generated new cluster ID: 7230d55bbed746a4955fb23656177f49
I20260926 20:52:38.803781 14540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:38.880883 14540 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:38.882328 14540 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:38.892546 14540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78e7ef1b0d1441cdbd9a8434ad2e67f9: Generated new TSK 0
I20260926 20:52:38.893445 14540 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:52:38.928318 14558 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 20:52:38.928462 14558 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:52:38.929749 14531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:38.909539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45014 (local address 127.8.249.190:34511)
0926 20:52:38.909729 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:38.909737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:38.910224 (+ 487us) server_negotiation.cc:407] Connection header received
0926 20:52:38.910415 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:38.910420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:38.910513 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:38.910684 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:52:38.911164 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:38.912327 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:38.913717 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:38.914381 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:38.914521 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:38.915009 (+ 488us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:52:38.929028 (+ 14019us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:52:38.929419 (+ 391us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45014: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 20:52:38.929785 14557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:52:38.909499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:34511 (local address 127.0.0.1:45014)
0926 20:52:38.910126 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:38.910142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:38.910259 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:38.910696 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:38.910700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:38.910709 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:52:38.911007 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:38.911015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:38.912486 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:38.912489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:38.913580 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:38.913590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:38.914555 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:38.914559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:38.914687 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:38.914718 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:52:38.929286 (+ 14568us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:52:38.929504 (+ 218us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.249.190:34511: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":347,"threads_started":1}
W20260926 20:52:38.929881 14531 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45014: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 20:52:38.931314 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 14462
I20260926 20:52:38.932482 14523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:39.026760 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 14462
2026-09-26T20:52:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (791 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 20:52:39.079468 9190 mini_oidc.cc:152] Starting JWKS server
I20260926 20:52:39.081377 9190 webserver.cc:460] Webserver started at https://127.0.0.1:46561/ using document root <none> and password file <none>
I20260926 20:52:39.081640 9190 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:52:39.082266 9190 webserver.cc:460] Webserver started at http://127.0.0.1:37835/ using document root <none> and password file <none>
2026-09-26T20:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:39Z Disabled control of system clock
I20260926 20:52:39.109211 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:38381
--webserver_interface=127.8.249.190
--webserver_port=0
--builtin_ntp_servers=127.8.249.148:42305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:38381
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46561/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:52:39.428406 14565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:39.428746 14565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:39.428921 14565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:39.436071 14565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:39.436225 14565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:39.436316 14565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:39.436401 14565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:39.436509 14565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:52:39.436606 14565 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:52:39.436683 14565 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46561/jwks.json
I20260926 20:52:39.445634 14565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:38381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:38381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46561/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:39.447050 14565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:39.448876 14565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:39.460641 14573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:39.460671 14570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:39.460841 14571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:39.461791 14565 server_base.cc:1034] running on GCE node
I20260926 20:52:39.462677 14565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:39.464210 14565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:39.465476 14565 hybrid_clock.cc:648] HybridClock initialized: now 1790455959465426 us; error 62 us; skew 500 ppm
I20260926 20:52:39.465900 14565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:52:39.468386 14565 webserver.cc:460] Webserver started at http://127.8.249.190:40875/ using document root <none> and password file <none>
I20260926 20:52:39.469105 14565 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:39.469220 14565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:39.469560 14565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:39.472497 14565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "1767629256ed46778e4c672100d8eae7"
format_stamp: "Formatted at 2026-09-26 20:52:39 on dist-test-slave-2dg3"
server_key: "336bd6134878edf057e719111227b8d9"
server_key_iv: "05f7e7ed4bec314bc96df876745d5f6d"
server_key_version: "encryptionkey@0"
I20260926 20:52:39.473268 14565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "1767629256ed46778e4c672100d8eae7"
format_stamp: "Formatted at 2026-09-26 20:52:39 on dist-test-slave-2dg3"
server_key: "336bd6134878edf057e719111227b8d9"
server_key_iv: "05f7e7ed4bec314bc96df876745d5f6d"
server_key_version: "encryptionkey@0"
I20260926 20:52:39.478351 14565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 20:52:39.483116 14579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:39.484822 14565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 20:52:39.485003 14565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "1767629256ed46778e4c672100d8eae7"
format_stamp: "Formatted at 2026-09-26 20:52:39 on dist-test-slave-2dg3"
server_key: "336bd6134878edf057e719111227b8d9"
server_key_iv: "05f7e7ed4bec314bc96df876745d5f6d"
server_key_version: "encryptionkey@0"
I20260926 20:52:39.485245 14565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:39.510152 14565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:39.511276 14565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:39.511622 14565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:39.537577 14565 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:38381
I20260926 20:52:39.537612 14630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:38381 every 8 connection(s)
I20260926 20:52:39.539063 14565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:39.542599 14631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:39.548744 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 14565
I20260926 20:52:39.548985 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:39.549767 9190 external_mini_cluster.cc:1428] Setting key 1941fc396252c7da7dcd333b380d92f3
I20260926 20:52:39.554256 14631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7: Bootstrap starting.
I20260926 20:52:39.558501 14631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:39.559927 14631 log.cc:824] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:39.562901 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:39.551607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.8.249.190:38381)
0926 20:52:39.552509 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:39.552518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:39.552555 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:52:39.554125 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:39.554146 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:39.554636 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:39.555057 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:39.555583 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:39.556893 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:39.558162 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:39.558984 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:39.559178 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:39.560348 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:39.560355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:39.560369 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:39.561158 (+ 789us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:39.561258 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:39.561285 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:39.561449 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:39.561666 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 20:52:39.561969 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":352,"threads_started":1}
I20260926 20:52:39.563215 14631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7: No bootstrap required, opened a new log
I20260926 20:52:39.566731 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:52:39.572510 14631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1767629256ed46778e4c672100d8eae7" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 38381 } }
I20260926 20:52:39.572891 14631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:39.573014 14631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1767629256ed46778e4c672100d8eae7, State: Initialized, Role: FOLLOWER
I20260926 20:52:39.573555 14631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1767629256ed46778e4c672100d8eae7" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 38381 } }
I20260926 20:52:39.573788 14631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:39.573937 14631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:39.574118 14631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:39.576421 14631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1767629256ed46778e4c672100d8eae7" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 38381 } }
I20260926 20:52:39.576951 14631 leader_election.cc:302] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1767629256ed46778e4c672100d8eae7; no voters:
I20260926 20:52:39.577621 14631 leader_election.cc:288] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:39.577780 14636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:39.578488 14636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [term 1 LEADER]: Becoming Leader. State: Replica: 1767629256ed46778e4c672100d8eae7, State: Running, Role: LEADER
I20260926 20:52:39.579123 14636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1767629256ed46778e4c672100d8eae7" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 38381 } }
I20260926 20:52:39.579900 14631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:39.582381 14637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1767629256ed46778e4c672100d8eae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1767629256ed46778e4c672100d8eae7" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 38381 } } }
I20260926 20:52:39.582880 14637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:39.583220 14638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1767629256ed46778e4c672100d8eae7. Latest consensus state: current_term: 1 leader_uuid: "1767629256ed46778e4c672100d8eae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1767629256ed46778e4c672100d8eae7" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 38381 } } }
I20260926 20:52:39.583530 14638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:39.584964 14647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:39.590171 14647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:39.599435 14647 catalog_manager.cc:1347] Generated new cluster ID: b6f01ce1a6954505b9a7c1fe4bd74eba
I20260926 20:52:39.599570 14647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:39.623070 14647 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:39.624372 14647 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:39.635918 14647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1767629256ed46778e4c672100d8eae7: Generated new TSK 0
I20260926 20:52:39.636740 14647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:52:39.660961 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:39.652158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34136 (local address 127.8.249.190:38381)
0926 20:52:39.652359 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:39.652365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:39.652741 (+ 376us) server_negotiation.cc:407] Connection header received
0926 20:52:39.652967 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:39.652973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:39.653080 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:39.653264 (+ 184us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:52:39.653683 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:39.654897 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:39.656354 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:39.657037 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:39.657170 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:39.657684 (+ 514us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:52:39.660266 (+ 2582us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46561/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:52:39.660632 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34136: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46561/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 20:52:39.661108 14634 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34136: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46561/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 20:52:39.662604 14660 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:52:39.652002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:38381 (local address 127.0.0.1:34136)
0926 20:52:39.652633 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:39.652649 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:39.652774 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:39.653276 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:39.653279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:39.653289 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:52:39.653535 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:39.653542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:39.655094 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:39.655099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:39.656194 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:39.656207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:39.657225 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:39.657230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:39.657361 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:39.657394 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:52:39.660520 (+ 3126us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46561/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:52:39.662175 (+ 1655us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.249.190:38381: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46561/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":1329,"thread_start_us":342,"threads_started":1}
I20260926 20:52:39.663653 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 14565
I20260926 20:52:39.664942 14626 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 20:52:39.672626 14661 webserver.cc:533] Webserver: sslize failed:
E20260926 20:52:39.672740 14661 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:52:39.756256 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 14565
2026-09-26T20:52:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (732 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T20:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:39Z Disabled control of system clock
I20260926 20:52:39.838335 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:43467
--webserver_interface=127.8.249.190
--webserver_port=0
--builtin_ntp_servers=127.8.249.148:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:43467
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/testchainca.cert with env {}
W20260926 20:52:40.156597 14666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:40.156889 14666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:40.157073 14666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:40.164371 14666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:40.164481 14666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:40.164520 14666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:40.164595 14666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:40.164662 14666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:52:40.164774 14666 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:52:40.164845 14666 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 20:52:40.175019 14666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:43467
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:43467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:40.176374 14666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:40.177686 14666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:40.189622 14671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:40.189673 14674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:40.189697 14672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:40.190603 14666 server_base.cc:1034] running on GCE node
I20260926 20:52:40.191413 14666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:40.192826 14666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:40.194056 14666 hybrid_clock.cc:648] HybridClock initialized: now 1790455960194000 us; error 51 us; skew 500 ppm
I20260926 20:52:40.194406 14666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:52:40.196709 14666 webserver.cc:460] Webserver started at http://127.8.249.190:40241/ using document root <none> and password file <none>
I20260926 20:52:40.197391 14666 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:40.197526 14666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:40.197834 14666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:40.200656 14666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "32f8bcb33acb4170838121b9505b458d"
format_stamp: "Formatted at 2026-09-26 20:52:40 on dist-test-slave-2dg3"
server_key: "6371ecf4e7b7df0ed5de8a02f58ea97b"
server_key_iv: "c48306541f5f4f73adfbe89df48ad88e"
server_key_version: "encryptionkey@0"
I20260926 20:52:40.201426 14666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "32f8bcb33acb4170838121b9505b458d"
format_stamp: "Formatted at 2026-09-26 20:52:40 on dist-test-slave-2dg3"
server_key: "6371ecf4e7b7df0ed5de8a02f58ea97b"
server_key_iv: "c48306541f5f4f73adfbe89df48ad88e"
server_key_version: "encryptionkey@0"
I20260926 20:52:40.206274 14666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 20:52:40.209852 14680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:40.211241 14666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 20:52:40.211439 14666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "32f8bcb33acb4170838121b9505b458d"
format_stamp: "Formatted at 2026-09-26 20:52:40 on dist-test-slave-2dg3"
server_key: "6371ecf4e7b7df0ed5de8a02f58ea97b"
server_key_iv: "c48306541f5f4f73adfbe89df48ad88e"
server_key_version: "encryptionkey@0"
I20260926 20:52:40.211658 14666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:40.228786 14666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:40.229708 14666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:40.230000 14666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:40.247793 14666 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:43467
I20260926 20:52:40.247799 14731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:43467 every 8 connection(s)
I20260926 20:52:40.249167 14666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:40.252897 14732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:40.257870 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 14666
I20260926 20:52:40.258101 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:40.258729 9190 external_mini_cluster.cc:1428] Setting key 495bc6decd9df524fff4a028dfa48351
I20260926 20:52:40.263892 14732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d: Bootstrap starting.
I20260926 20:52:40.266898 14732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:40.267911 14732 log.cc:824] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:40.271178 14732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d: No bootstrap required, opened a new log
I20260926 20:52:40.272334 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:40.260692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38426 (local address 127.8.249.190:43467)
0926 20:52:40.261725 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:40.261737 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:40.261780 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:52:40.263490 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:40.263513 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:40.263956 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:40.264404 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:40.264943 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:40.266206 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:40.267573 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:40.268305 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:40.268549 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:40.269645 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:40.269655 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:40.269671 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:52:40.270341 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:40.270438 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:52:40.270462 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:40.270579 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:40.270780 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 20:52:40.271109 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":339,"threads_started":1}
I20260926 20:52:40.276121 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:52:40.279945 14732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f8bcb33acb4170838121b9505b458d" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43467 } }
I20260926 20:52:40.280319 14732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:40.280423 14732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32f8bcb33acb4170838121b9505b458d, State: Initialized, Role: FOLLOWER
I20260926 20:52:40.280975 14732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f8bcb33acb4170838121b9505b458d" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43467 } }
I20260926 20:52:40.281203 14732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:40.281359 14732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:40.281538 14732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:40.283730 14732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f8bcb33acb4170838121b9505b458d" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43467 } }
I20260926 20:52:40.284216 14732 leader_election.cc:302] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32f8bcb33acb4170838121b9505b458d; no voters:
I20260926 20:52:40.284852 14732 leader_election.cc:288] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:40.285050 14737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:40.286275 14737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [term 1 LEADER]: Becoming Leader. State: Replica: 32f8bcb33acb4170838121b9505b458d, State: Running, Role: LEADER
I20260926 20:52:40.286932 14737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f8bcb33acb4170838121b9505b458d" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43467 } }
I20260926 20:52:40.287220 14732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:40.290252 14739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32f8bcb33acb4170838121b9505b458d. Latest consensus state: current_term: 1 leader_uuid: "32f8bcb33acb4170838121b9505b458d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f8bcb33acb4170838121b9505b458d" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43467 } } }
I20260926 20:52:40.290333 14738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32f8bcb33acb4170838121b9505b458d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f8bcb33acb4170838121b9505b458d" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43467 } } }
I20260926 20:52:40.290642 14739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:40.290666 14738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:40.297204 14747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:40.301319 14747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:40.309422 14747 catalog_manager.cc:1347] Generated new cluster ID: 355c13e674004a978a7ccd3284f47a5a
I20260926 20:52:40.309547 14747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:40.366780 14747 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:40.368639 14747 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:40.376780 14747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32f8bcb33acb4170838121b9505b458d: Generated new TSK 0
I20260926 20:52:40.377779 14747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:52:40.421558 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:40.394733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.8.249.190:43467)
0926 20:52:40.394944 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:40.394951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:40.395290 (+ 339us) server_negotiation.cc:407] Connection header received
0926 20:52:40.395521 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:40.395527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:40.395625 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:40.395790 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:52:40.396210 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:40.397337 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:40.398642 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:40.399391 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:40.399526 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:40.399955 (+ 429us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:52:40.420800 (+ 20845us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:52:40.421202 (+ 402us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38434: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":93}
W20260926 20:52:40.421623 14761 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:52:40.394591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:43467 (local address 127.0.0.1:38434)
0926 20:52:40.395192 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:40.395209 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:40.395324 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:40.395803 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:40.395806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:40.395815 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:52:40.396076 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:40.396084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:40.397474 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:40.397478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:40.398505 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:40.398515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:40.399541 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:40.399545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:40.399677 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:40.399706 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:52:40.421087 (+ 21381us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:52:40.421339 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.249.190:43467: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":327,"threads_started":1}
W20260926 20:52:40.421698 14735 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38434: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260926 20:52:40.422719 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 14666
I20260926 20:52:40.423907 14727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:40.521556 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 14666
2026-09-26T20:52:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (741 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (407 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (407 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:52:41 dist-test-slave-2dg3 krb5kdc[14776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:52:41 dist-test-slave-2dg3 krb5kdc[14776](info): set up 2 sockets
Sep 26 20:52:41 dist-test-slave-2dg3 krb5kdc[14776](info): commencing operation
krb5kdc: starting...
W20260926 20:52:42.415373 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:52:42 dist-test-slave-2dg3 krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:52:42.620370 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:41267
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:41267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:42.939677 14789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:42.939966 14789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:42.940057 14789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:42.947088 14789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:42.947206 14789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:42.947252 14789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:42.947322 14789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:42.947359 14789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:42.957015 14789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:41267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:41267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:42.958253 14789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:42.959481 14789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:42.971097 14795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:42.971172 14794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:42.971187 14797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:42.971743 14789 server_base.cc:1034] running on GCE node
I20260926 20:52:42.972500 14789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:42.973847 14789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:42.975032 14789 hybrid_clock.cc:648] HybridClock initialized: now 1790455962974987 us; error 30 us; skew 500 ppm
I20260926 20:52:42.975370 14789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:42 dist-test-slave-2dg3 krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455962, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:42.978325 14789 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:52:42.979987 14789 webserver.cc:460] Webserver started at http://127.8.249.190:37655/ using document root <none> and password file <none>
I20260926 20:52:42.980619 14789 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:42.980716 14789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:42.980981 14789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:42.983892 14789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "0d58c25c5ffc4621b91bbe7bfd3b91bb"
format_stamp: "Formatted at 2026-09-26 20:52:42 on dist-test-slave-2dg3"
server_key: "9233d154535c42ab3c05ad1e144b615a"
server_key_iv: "81189ab05e03f3521e9a10811b34bd57"
server_key_version: "encryptionkey@0"
I20260926 20:52:42.984650 14789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "0d58c25c5ffc4621b91bbe7bfd3b91bb"
format_stamp: "Formatted at 2026-09-26 20:52:42 on dist-test-slave-2dg3"
server_key: "9233d154535c42ab3c05ad1e144b615a"
server_key_iv: "81189ab05e03f3521e9a10811b34bd57"
server_key_version: "encryptionkey@0"
I20260926 20:52:42.989643 14789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 20:52:42.992865 14804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:42.994220 14789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:52:42.994400 14789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "0d58c25c5ffc4621b91bbe7bfd3b91bb"
format_stamp: "Formatted at 2026-09-26 20:52:42 on dist-test-slave-2dg3"
server_key: "9233d154535c42ab3c05ad1e144b615a"
server_key_iv: "81189ab05e03f3521e9a10811b34bd57"
server_key_version: "encryptionkey@0"
I20260926 20:52:42.994593 14789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:43.028000 14789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:43.032092 14789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:43.032389 14789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:43.050657 14789 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:41267
I20260926 20:52:43.050679 14855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:41267 every 8 connection(s)
I20260926 20:52:43.052089 14789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:43.055876 14856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:43.060942 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 14789
I20260926 20:52:43.061170 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:43.061935 9190 external_mini_cluster.cc:1428] Setting key b819fb7e79766881162f87343e614b70
I20260926 20:52:43.067739 14856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb: Bootstrap starting.
I20260926 20:52:43.070924 14856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:43.071949 14856 log.cc:824] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:43.074945 14856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb: No bootstrap required, opened a new log
Sep 26 20:52:43 dist-test-slave-2dg3 krb5kdc[14776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:43.083067 14856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d58c25c5ffc4621b91bbe7bfd3b91bb" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41267 } }
I20260926 20:52:43.083441 14856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:43.083576 14856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d58c25c5ffc4621b91bbe7bfd3b91bb, State: Initialized, Role: FOLLOWER
I20260926 20:52:43.084100 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:43.064143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42212 (local address 127.8.249.190:41267)
0926 20:52:43.064938 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:43.064953 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:43.064996 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:52:43.066695 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:43.066719 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:43.067210 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:43.067614 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:43.069054 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:43.070356 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:43.071725 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:43.072562 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:43.072765 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:43.076838 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:43.076846 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:43.076860 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:43.076900 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:43.079929 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:43.080599 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:43.080614 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:43.080620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:43.080784 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:43.081180 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:43.081185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:43.081194 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:43.081706 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:43.082022 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:43.082446 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 20:52:43.082909 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":325,"threads_started":1}
I20260926 20:52:43.084174 14856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d58c25c5ffc4621b91bbe7bfd3b91bb" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41267 } }
I20260926 20:52:43.084491 14856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:43.084656 14856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:43.084856 14856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:43.087218 14856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d58c25c5ffc4621b91bbe7bfd3b91bb" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41267 } }
I20260926 20:52:43.087704 14856 leader_election.cc:302] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d58c25c5ffc4621b91bbe7bfd3b91bb; no voters:
I20260926 20:52:43.088276 14856 leader_election.cc:288] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:43.088402 14861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:43.089085 14861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [term 1 LEADER]: Becoming Leader. State: Replica: 0d58c25c5ffc4621b91bbe7bfd3b91bb, State: Running, Role: LEADER
I20260926 20:52:43.089583 14861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d58c25c5ffc4621b91bbe7bfd3b91bb" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41267 } }
I20260926 20:52:43.090269 14856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:43.092941 14863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d58c25c5ffc4621b91bbe7bfd3b91bb. Latest consensus state: current_term: 1 leader_uuid: "0d58c25c5ffc4621b91bbe7bfd3b91bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d58c25c5ffc4621b91bbe7bfd3b91bb" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41267 } } }
I20260926 20:52:43.093559 14863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:43.093024 14862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d58c25c5ffc4621b91bbe7bfd3b91bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d58c25c5ffc4621b91bbe7bfd3b91bb" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41267 } } }
I20260926 20:52:43.094112 14862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:43.098722 14868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:43.103753 14868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:43.112542 14868 catalog_manager.cc:1347] Generated new cluster ID: 025cf308f1e74d2c9f751510761e0514
I20260926 20:52:43.112670 14868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:43.125895 14868 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:43.127702 14868 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:43.142230 14868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d58c25c5ffc4621b91bbe7bfd3b91bb: Generated new TSK 0
I20260926 20:52:43.143127 14868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:52:43.202008 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:41267
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:43.539753 14882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:43.540052 14882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:43.540208 14882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:43.547508 14882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:43.547637 14882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:43.547819 14882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:52:43.557603 14882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:41267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:43.558992 14882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:43.560346 14882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:43.573350 14888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:43.573323 14887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:43.573549 14890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:43.574128 14882 server_base.cc:1034] running on GCE node
I20260926 20:52:43.574672 14882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:43.575701 14882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:43.576898 14882 hybrid_clock.cc:648] HybridClock initialized: now 1790455963576859 us; error 29 us; skew 500 ppm
I20260926 20:52:43.577209 14882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:43 dist-test-slave-2dg3 krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455963, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:43.580288 14882 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:52:43.581712 14882 webserver.cc:460] Webserver started at http://127.8.249.129:45721/ using document root <none> and password file <none>
I20260926 20:52:43.582379 14882 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:43.582510 14882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:43.582904 14882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:43.586699 14882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "79ed0042a1d14148bf30b54f5529b2bc"
format_stamp: "Formatted at 2026-09-26 20:52:43 on dist-test-slave-2dg3"
server_key: "cc49de6b9d7e54501f683408e29f7944"
server_key_iv: "28b4da49d1bcc28dfd35a7de15b13609"
server_key_version: "encryptionkey@0"
I20260926 20:52:43.587782 14882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "79ed0042a1d14148bf30b54f5529b2bc"
format_stamp: "Formatted at 2026-09-26 20:52:43 on dist-test-slave-2dg3"
server_key: "cc49de6b9d7e54501f683408e29f7944"
server_key_iv: "28b4da49d1bcc28dfd35a7de15b13609"
server_key_version: "encryptionkey@0"
I20260926 20:52:43.593035 14882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:52:43.596637 14897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:43.597947 14882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:52:43.598114 14882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "79ed0042a1d14148bf30b54f5529b2bc"
format_stamp: "Formatted at 2026-09-26 20:52:43 on dist-test-slave-2dg3"
server_key: "cc49de6b9d7e54501f683408e29f7944"
server_key_iv: "28b4da49d1bcc28dfd35a7de15b13609"
server_key_version: "encryptionkey@0"
I20260926 20:52:43.598354 14882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:43.611928 14882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:43.615700 14882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:43.615943 14882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:43.616838 14882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:43.618069 14882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:43.618139 14882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:43.618220 14882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:43.618269 14882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:43.651218 14882 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:46607
I20260926 20:52:43.651263 15009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:46607 every 8 connection(s)
I20260926 20:52:43.652472 14882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:52:43.657469 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 14882
I20260926 20:52:43.657677 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:52:43.658442 9190 external_mini_cluster.cc:1428] Setting key e663f441b7547e7a35421e22c8b5536e
Sep 26 20:52:43 dist-test-slave-2dg3 krb5kdc[14776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455963, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:43.675601 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:43.654970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:48993 (local address 127.8.249.190:41267)
0926 20:52:43.655390 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:43.655394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:43.656411 (+ 1017us) server_negotiation.cc:407] Connection header received
0926 20:52:43.658857 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:43.658861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:43.658947 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:43.659043 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:43.660883 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:43.661615 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:43.664143 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:43.664537 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:43.664635 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:43.668792 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:43.668801 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:43.668810 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:43.668865 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:43.672261 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:43.672971 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:43.672980 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:43.673003 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:43.673147 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:43.673580 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:43.673587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:43.673593 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:43.673984 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:43.674129 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:43.674861 (+ 732us) server_negotiation.cc:299] Negotiation successful
0926 20:52:43.675120 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":215,"threads_started":1}
I20260926 20:52:43.676260 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:43.655552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:41267 (local address 127.8.249.129:48993)
0926 20:52:43.656254 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:43.656296 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:43.658512 (+ 2216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:43.659282 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:43.659290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:43.659926 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:43.660666 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:43.660692 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:43.661779 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:43.661788 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:43.663971 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:43.663988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:43.664768 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:43.664779 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:43.665112 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:43.665898 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:43.665932 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:43.668558 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:43.672396 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:43.672402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:43.672416 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:43.672836 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:43.673258 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:43.673261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:43.673263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:43.673443 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:43.674104 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:43.674110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:43.674441 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 20:52:43.674656 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 20:52:43.675124 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":277,"threads_started":1}
I20260926 20:52:43.677891 15010 heartbeater.cc:347] Connected to a master server at 127.8.249.190:41267
I20260926 20:52:43.678268 15010 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:43.679271 15010 heartbeater.cc:511] Master 127.8.249.190:41267 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
I20260926 20:52:43.681860 14821 ts_manager.cc:194] Registered new tserver with Master: 79ed0042a1d14148bf30b54f5529b2bc (127.8.249.129:46607)
I20260926 20:52:43.683427 14821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:48993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:52:43.709880 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:41267
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:44.029417 15016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:44.029738 15016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:44.029917 15016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:44.037225 15016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:44.037393 15016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:44.037571 15016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:52:44.047435 15016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:41267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:44.048861 15016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:44.050282 15016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:44.064343 15022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:44.064929 15024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:44.064872 15021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:44.065357 15016 server_base.cc:1034] running on GCE node
I20260926 20:52:44.066015 15016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:44.067059 15016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:44.068286 15016 hybrid_clock.cc:648] HybridClock initialized: now 1790455964068225 us; error 53 us; skew 500 ppm
I20260926 20:52:44.068609 15016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:44.075667 15016 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:52:44.077293 15016 webserver.cc:460] Webserver started at http://127.8.249.130:39923/ using document root <none> and password file <none>
I20260926 20:52:44.078015 15016 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:44.078152 15016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:44.078471 15016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:44.081391 15016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "3e00301e5e104e1bb076076b3205bbef"
format_stamp: "Formatted at 2026-09-26 20:52:44 on dist-test-slave-2dg3"
server_key: "b4076a5b88b2575083b5f52b9f05ae2c"
server_key_iv: "9ea005f7dbc33e11630070b68cba2b2b"
server_key_version: "encryptionkey@0"
I20260926 20:52:44.082155 15016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "3e00301e5e104e1bb076076b3205bbef"
format_stamp: "Formatted at 2026-09-26 20:52:44 on dist-test-slave-2dg3"
server_key: "b4076a5b88b2575083b5f52b9f05ae2c"
server_key_iv: "9ea005f7dbc33e11630070b68cba2b2b"
server_key_version: "encryptionkey@0"
I20260926 20:52:44.087138 15016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 20:52:44.090457 15031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:44.091903 15016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:44.092041 15016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "3e00301e5e104e1bb076076b3205bbef"
format_stamp: "Formatted at 2026-09-26 20:52:44 on dist-test-slave-2dg3"
server_key: "b4076a5b88b2575083b5f52b9f05ae2c"
server_key_iv: "9ea005f7dbc33e11630070b68cba2b2b"
server_key_version: "encryptionkey@0"
I20260926 20:52:44.092209 15016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:44.111084 15016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:44.115329 15016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:44.115568 15016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:44.116460 15016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:44.117739 15016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:44.117810 15016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:44.117893 15016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:44.117936 15016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:44.149410 15016 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:39609
I20260926 20:52:44.149456 15143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:39609 every 8 connection(s)
I20260926 20:52:44.150703 15016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:52:44.160127 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 15016
I20260926 20:52:44.160319 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:52:44.160764 9190 external_mini_cluster.cc:1428] Setting key 9e2d4071a2987d7aa99fdf01b52f8406
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:44.170974 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:44.153295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:54275 (local address 127.8.249.190:41267)
0926 20:52:44.153481 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:44.153485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:44.154786 (+ 1301us) server_negotiation.cc:407] Connection header received
0926 20:52:44.156590 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:44.156594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:44.156663 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:44.156765 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:44.158399 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.159113 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.160751 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.161217 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.161354 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:44.164822 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:44.164826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:44.164829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:44.164861 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:44.167498 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.168423 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.168431 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.168437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.168604 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.169064 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.169070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.169075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.169437 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:44.169611 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:44.170252 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 20:52:44.170506 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:52:44.171727 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:44.153772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:41267 (local address 127.8.249.130:54275)
0926 20:52:44.154620 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:44.154664 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:44.156347 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:44.156975 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:44.156983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:44.157473 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:44.158216 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:44.158234 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.159247 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.159251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:44.160610 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:44.160621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.161440 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.161447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:44.161712 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:44.162476 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:44.162502 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:44.164602 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:44.167696 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:44.167706 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:44.167723 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:44.168291 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:44.168749 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:44.168755 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:44.168757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:44.168950 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:44.169562 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:44.169569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:44.169892 (+ 323us) client_negotiation.cc:769] Sending connection context
0926 20:52:44.170134 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 20:52:44.170539 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":351,"threads_started":1}
I20260926 20:52:44.173110 15144 heartbeater.cc:347] Connected to a master server at 127.8.249.190:41267
I20260926 20:52:44.173424 15144 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:44.174293 15144 heartbeater.cc:511] Master 127.8.249.190:41267 requested a full tablet report, sending...
I20260926 20:52:44.175945 14821 ts_manager.cc:194] Registered new tserver with Master: 3e00301e5e104e1bb076076b3205bbef (127.8.249.130:39609)
I20260926 20:52:44.176834 14821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:54275
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:52:44.209983 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:41267
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:44.526684 15149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:44.527050 15149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:44.527225 15149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:44.534405 15149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:44.534587 15149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:44.534804 15149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:52:44.544513 15149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:41267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:44.545900 15149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:44.547256 15149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:44.560575 15154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:44.560619 15157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:44.560585 15155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:44.561230 15149 server_base.cc:1034] running on GCE node
I20260926 20:52:44.561784 15149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:44.562628 15149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:44.563822 15149 hybrid_clock.cc:648] HybridClock initialized: now 1790455964563789 us; error 66 us; skew 500 ppm
I20260926 20:52:44.564136 15149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:44.567126 15149 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:52:44.568581 15149 webserver.cc:460] Webserver started at http://127.8.249.131:34637/ using document root <none> and password file <none>
I20260926 20:52:44.569308 15149 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:44.569408 15149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:44.569660 15149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:44.572435 15149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "217effe0a17347b8941b91764e3991cb"
format_stamp: "Formatted at 2026-09-26 20:52:44 on dist-test-slave-2dg3"
server_key: "9485c1035efb7ef3809298fb60221091"
server_key_iv: "7d256766c0a96a070f1d4a15355c3c48"
server_key_version: "encryptionkey@0"
I20260926 20:52:44.573145 15149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "217effe0a17347b8941b91764e3991cb"
format_stamp: "Formatted at 2026-09-26 20:52:44 on dist-test-slave-2dg3"
server_key: "9485c1035efb7ef3809298fb60221091"
server_key_iv: "7d256766c0a96a070f1d4a15355c3c48"
server_key_version: "encryptionkey@0"
I20260926 20:52:44.577948 15149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 20:52:44.581167 15164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:44.582449 15149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:52:44.582623 15149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "217effe0a17347b8941b91764e3991cb"
format_stamp: "Formatted at 2026-09-26 20:52:44 on dist-test-slave-2dg3"
server_key: "9485c1035efb7ef3809298fb60221091"
server_key_iv: "7d256766c0a96a070f1d4a15355c3c48"
server_key_version: "encryptionkey@0"
I20260926 20:52:44.582840 15149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:44.596228 15149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:44.600422 15149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:44.600719 15149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:44.601701 15149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:44.603157 15149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:44.603250 15149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:44.603394 15149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:44.603462 15149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:44.635259 15149 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:42679
I20260926 20:52:44.635349 15276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:42679 every 8 connection(s)
I20260926 20:52:44.636497 15149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:52:44.639562 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 15149
I20260926 20:52:44.639742 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:52:44.640362 9190 external_mini_cluster.cc:1428] Setting key beafeb2974d154d9aab8b2d14a083abb
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:52:44.656975 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:44.638895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:42421 (local address 127.8.249.190:41267)
0926 20:52:44.639059 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:44.639063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:44.640366 (+ 1303us) server_negotiation.cc:407] Connection header received
0926 20:52:44.642123 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:44.642129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:44.642251 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:44.642361 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:44.643818 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.644557 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.646102 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.646533 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.646669 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:44.649948 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:44.649954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:44.649960 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:44.650000 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:44.653280 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.654288 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.654295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.654300 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.654461 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.655023 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.655028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.655032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.655424 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:44.655600 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:44.656397 (+ 797us) server_negotiation.cc:299] Negotiation successful
0926 20:52:44.656621 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:52:44.658000 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:44.639423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:41267 (local address 127.8.249.131:42421)
0926 20:52:44.640203 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:44.640246 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:44.641831 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:44.642596 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:44.642604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:44.643102 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:44.643653 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:44.643669 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.644703 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.644709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:44.645957 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:44.645968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.646755 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.646763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:44.647021 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:44.647633 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:44.647654 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:44.649738 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:44.653492 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:44.653502 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:44.653519 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:44.654142 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:44.654589 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:44.654593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:44.654597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:44.654899 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:44.655555 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:44.655564 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:44.656015 (+ 451us) client_negotiation.cc:769] Sending connection context
0926 20:52:44.656279 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 20:52:44.656818 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":338,"threads_started":1}
I20260926 20:52:44.659415 15277 heartbeater.cc:347] Connected to a master server at 127.8.249.190:41267
I20260926 20:52:44.659734 15277 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:44.660444 15277 heartbeater.cc:511] Master 127.8.249.190:41267 requested a full tablet report, sending...
I20260926 20:52:44.661818 14821 ts_manager.cc:194] Registered new tserver with Master: 217effe0a17347b8941b91764e3991cb (127.8.249.131:42679)
I20260926 20:52:44.662588 14821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:42421
I20260926 20:52:44.667979 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:52:44.686565 15010 heartbeater.cc:503] Master 127.8.249.190:41267 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/test-admin.keytab.
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:44.693675 9190 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 20:52:44.699041 15284 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:52:44.696353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47976 (local address 127.8.249.129:46607)
0926 20:52:44.697193 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:44.697201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:44.697233 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:52:44.697310 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:44.697317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:44.697531 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:44.697699 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:44.698583 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47976: BlockingRecv error: recv got EOF from 127.0.0.1:47976 (error 108)
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":462,"threads_started":1}
W20260926 20:52:44.699546 9190 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/krb5cc)
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:44.701038 9190 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:44.702405 9190 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:52:44.716905 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:44.704073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.8.249.129:46607)
0926 20:52:44.704242 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:44.704246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:44.704396 (+ 150us) server_negotiation.cc:407] Connection header received
0926 20:52:44.704478 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:44.704482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:44.704540 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:44.704623 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:44.705566 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.706539 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.707764 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.708403 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.708525 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:44.711981 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:44.711992 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:44.712007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:44.712048 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:44.714544 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.715217 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.715223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.715226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.715335 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.715667 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.715670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.715672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.715993 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:44.716210 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:44.716470 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 20:52:44.716659 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 20:52:44.722301 15284 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:52:44.720819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.8.249.129:46607)
0926 20:52:44.721049 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:44.721054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:44.721223 (+ 169us) server_negotiation.cc:407] Connection header received
0926 20:52:44.721328 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:44.721331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:44.721394 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:44.721467 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:44.722071 (+ 604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47984: BlockingRecv error: recv got EOF from 127.0.0.1:47984 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 20:52:44.722440 9190 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/krb5cc)
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:44.723984 9190 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:44.725214 9190 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:52:44.738572 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:44.726487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47986 (local address 127.8.249.129:46607)
0926 20:52:44.726682 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:44.726688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:44.726904 (+ 216us) server_negotiation.cc:407] Connection header received
0926 20:52:44.727057 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:44.727068 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:44.727120 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:44.727219 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:44.728135 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.728825 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.730018 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.730431 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.730562 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:44.733908 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:44.733912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:44.733916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:44.733956 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:44.736488 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.737047 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.737051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.737053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.737161 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.737461 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.737464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.737466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.737701 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:44.737881 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:44.738110 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 20:52:44.738285 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 20:52:44.742460 15284 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:52:44.740953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.8.249.129:46607)
0926 20:52:44.741111 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:44.741115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:44.741420 (+ 305us) server_negotiation.cc:407] Connection header received
0926 20:52:44.741536 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:44.741539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:44.741600 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:44.741675 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:44.742229 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47998: BlockingRecv error: recv got EOF from 127.0.0.1:47998 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 20:52:44.742676 9190 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790455916331309-9190-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:44.744143 9190 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:44.745361 9190 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:52:44.759047 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:44.746502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.8.249.129:46607)
0926 20:52:44.746696 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:44.746703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:44.747017 (+ 314us) server_negotiation.cc:407] Connection header received
0926 20:52:44.747198 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:44.747203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:44.747265 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:44.747358 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:44.748296 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.748989 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.750155 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.750566 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.750692 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:44.754028 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:44.754034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:44.754038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:44.754076 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:44.756627 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.757240 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.757247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.757255 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.757395 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.757737 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.757743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.757746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.757985 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:44.758262 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:44.758534 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 20:52:44.758733 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:52:44.760545 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 14882
I20260926 20:52:44.771898 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:44.761213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.8.249.129:46607)
0926 20:52:44.761446 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:44.761451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:44.761656 (+ 205us) server_negotiation.cc:407] Connection header received
0926 20:52:44.761752 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:44.761755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:44.761805 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:44.761885 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:44.762734 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.763435 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.764547 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.764967 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.765057 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:44.767091 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:44.767096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:44.767099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:44.767128 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:44.769470 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.770060 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.770065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.770067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.770222 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.770526 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.770529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.770531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.770861 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:44.771075 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:44.771354 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 20:52:44.771579 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:52:44.772447 15005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:44.854434 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 14882
I20260926 20:52:44.870631 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 15016
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:52:44.885294 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:44.871522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.8.249.130:39609)
0926 20:52:44.872070 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:44.872077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:44.872103 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:52:44.872163 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:44.872170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:44.872398 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:44.872535 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:44.873416 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.874371 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.875563 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.876104 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.876202 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:44.879843 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:44.879854 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:44.879869 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:44.879904 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:44.882938 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.883587 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.883595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.883598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.883725 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.884027 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.884030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.884032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:44.884355 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:44.884580 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:44.884791 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 20:52:44.885009 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":269,"threads_started":1}
I20260926 20:52:44.886656 15139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:44.969734 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 15016
I20260926 20:52:44.986140 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 15149
Sep 26 20:52:44 dist-test-slave-2dg3 krb5kdc[14776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:52:45.001596 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:44.987225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47388 (local address 127.8.249.131:42679)
0926 20:52:44.987831 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:44.987838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:44.987867 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:52:44.987924 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:44.987930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:44.988134 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:44.988255 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:44.989257 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.990248 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.991810 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:44.992343 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:44.992437 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:44.996310 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:44.996321 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:44.996338 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:44.996368 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:44.999268 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:44.999896 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:44.999903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:44.999905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:45.000022 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:45.000340 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:45.000343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:45.000345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:45.000665 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:45.000912 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:45.001118 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 20:52:45.001314 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":335,"threads_started":1}
I20260926 20:52:45.002990 15272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:45.091647 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 15149
I20260926 20:52:45.107410 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 14789
I20260926 20:52:45.108567 14851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:45.210566 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 14789
2026-09-26T20:52:45Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3885 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:52:45 dist-test-slave-2dg3 krb5kdc[15304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:52:45 dist-test-slave-2dg3 krb5kdc[15304](info): set up 2 sockets
Sep 26 20:52:45 dist-test-slave-2dg3 krb5kdc[15304](info): commencing operation
krb5kdc: starting...
W20260926 20:52:46.295691 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:52:46 dist-test-slave-2dg3 krb5kdc[15304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:52:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.190.keytab.
Entry for principal oryx/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.190.keytab.
I20260926 20:52:46.510648 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:44135
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.190.keytab
--principal=oryx/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:44135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:46.832935 15317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:46.833263 15317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:46.833446 15317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:46.840934 15317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:46.841050 15317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:46.841150 15317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:46.841249 15317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:46.841331 15317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:46.851370 15317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:44135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.190.keytab
--principal=oryx/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:44135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:46.852638 15317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:46.853942 15317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:46.865486 15322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:46.865399 15323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:46.865446 15325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:46.866514 15317 server_base.cc:1034] running on GCE node
I20260926 20:52:46.867329 15317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:46.868690 15317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:46.869879 15317 hybrid_clock.cc:648] HybridClock initialized: now 1790455966869839 us; error 30 us; skew 500 ppm
I20260926 20:52:46.870252 15317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:46 dist-test-slave-2dg3 krb5kdc[15304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455966, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:46.873147 15317 init.cc:377] Logged in from keytab as oryx/127.8.249.190@KRBTEST.COM (short username oryx)
I20260926 20:52:46.874423 15317 webserver.cc:460] Webserver started at http://127.8.249.190:46813/ using document root <none> and password file <none>
I20260926 20:52:46.875108 15317 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:46.875257 15317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:46.875558 15317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:46.878378 15317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "9d6c916c8ec441e6be3173bb0f696d38"
format_stamp: "Formatted at 2026-09-26 20:52:46 on dist-test-slave-2dg3"
server_key: "29381fac56915ea36e3adf0a2b91b2ad"
server_key_iv: "7f31ad2f037866cc10eeb17378f35c35"
server_key_version: "encryptionkey@0"
I20260926 20:52:46.879151 15317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "9d6c916c8ec441e6be3173bb0f696d38"
format_stamp: "Formatted at 2026-09-26 20:52:46 on dist-test-slave-2dg3"
server_key: "29381fac56915ea36e3adf0a2b91b2ad"
server_key_iv: "7f31ad2f037866cc10eeb17378f35c35"
server_key_version: "encryptionkey@0"
I20260926 20:52:46.883983 15317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:52:46.887315 15332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:46.888646 15317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:52:46.888824 15317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "9d6c916c8ec441e6be3173bb0f696d38"
format_stamp: "Formatted at 2026-09-26 20:52:46 on dist-test-slave-2dg3"
server_key: "29381fac56915ea36e3adf0a2b91b2ad"
server_key_iv: "7f31ad2f037866cc10eeb17378f35c35"
server_key_version: "encryptionkey@0"
I20260926 20:52:46.888998 15317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:46.912565 15317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:46.916605 15317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:46.916899 15317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:46.934549 15317 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:44135
I20260926 20:52:46.934588 15383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:44135 every 8 connection(s)
I20260926 20:52:46.935861 15317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:46.939528 15384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:46.940187 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 15317
I20260926 20:52:46.940390 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:46.941172 9190 external_mini_cluster.cc:1428] Setting key 031235867cbb74894410f52001bb9887
I20260926 20:52:46.954849 15384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38: Bootstrap starting.
Sep 26 20:52:46 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:52:46.957888 15384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:46.959369 15384 log.cc:824] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:46.962854 15384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38: No bootstrap required, opened a new log
I20260926 20:52:46.966686 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:46.943303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.8.249.190:44135)
0926 20:52:46.944216 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:46.944227 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:46.944273 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:52:46.946706 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:46.946743 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:46.947366 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:46.947907 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:46.949815 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:46.951918 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:46.953274 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:46.954282 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:46.954462 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:46.958265 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:46.958277 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:46.958302 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:46.958354 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:46.962324 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:46.963045 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:46.963053 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:46.963060 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:46.963270 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:46.963675 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:46.963682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:46.963689 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:46.964309 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:46.964618 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:46.964909 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 20:52:46.965357 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":476,"threads_started":1}
I20260926 20:52:46.972019 15384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } }
I20260926 20:52:46.972342 15384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:46.972412 15384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6c916c8ec441e6be3173bb0f696d38, State: Initialized, Role: FOLLOWER
I20260926 20:52:46.972965 15384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } }
I20260926 20:52:46.973167 15384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:46.973251 15384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:46.973376 15384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:46.975780 15384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } }
I20260926 20:52:46.976236 15384 leader_election.cc:302] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d6c916c8ec441e6be3173bb0f696d38; no voters:
I20260926 20:52:46.976758 15384 leader_election.cc:288] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:46.976912 15389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:46.977676 15389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [term 1 LEADER]: Becoming Leader. State: Replica: 9d6c916c8ec441e6be3173bb0f696d38, State: Running, Role: LEADER
I20260926 20:52:46.978229 15389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } }
I20260926 20:52:46.979110 15384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:46.981235 15390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } }
I20260926 20:52:46.981604 15390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:46.981199 15391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d6c916c8ec441e6be3173bb0f696d38. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } }
I20260926 20:52:46.982455 15391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:46.983996 15395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:46.989744 15395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:46.998260 15395 catalog_manager.cc:1347] Generated new cluster ID: 986a280ad246442aafa35051eae5c0c7
I20260926 20:52:46.998371 15395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:47.007352 15395 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:47.008689 15395 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:47.014611 15395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38: Generated new TSK 0
I20260926 20:52:47.015434 15395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab.
Entry for principal oryx/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab.
I20260926 20:52:47.084246 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44135
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab
--principal=oryx/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:47.452723 15410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:47.453053 15410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:47.453217 15410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:47.460561 15410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:47.460667 15410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:47.460819 15410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:52:47.470873 15410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab
--principal=oryx/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:47.472172 15410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:47.473490 15410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:47.486574 15415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:47.486574 15416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:47.486486 15418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:47.487399 15410 server_base.cc:1034] running on GCE node
I20260926 20:52:47.488009 15410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:47.488927 15410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:47.490146 15410 hybrid_clock.cc:648] HybridClock initialized: now 1790455967490088 us; error 51 us; skew 500 ppm
I20260926 20:52:47.490442 15410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:47 dist-test-slave-2dg3 krb5kdc[15304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455967, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:47.493589 15410 init.cc:377] Logged in from keytab as oryx/127.8.249.129@KRBTEST.COM (short username oryx)
I20260926 20:52:47.494966 15410 webserver.cc:460] Webserver started at http://127.8.249.129:40951/ using document root <none> and password file <none>
I20260926 20:52:47.495589 15410 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:47.495682 15410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:47.495918 15410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:47.498603 15410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "db22faab4b5d414fadbce62b9eec0d9e"
format_stamp: "Formatted at 2026-09-26 20:52:47 on dist-test-slave-2dg3"
server_key: "7618b31b8ef3ac326ce5d1d302047625"
server_key_iv: "209b4b94a13989f418129f23c97b1c25"
server_key_version: "encryptionkey@0"
I20260926 20:52:47.499331 15410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "db22faab4b5d414fadbce62b9eec0d9e"
format_stamp: "Formatted at 2026-09-26 20:52:47 on dist-test-slave-2dg3"
server_key: "7618b31b8ef3ac326ce5d1d302047625"
server_key_iv: "209b4b94a13989f418129f23c97b1c25"
server_key_version: "encryptionkey@0"
I20260926 20:52:47.504138 15410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 20:52:47.507310 15425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:47.508589 15410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 20:52:47.508754 15410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "db22faab4b5d414fadbce62b9eec0d9e"
format_stamp: "Formatted at 2026-09-26 20:52:47 on dist-test-slave-2dg3"
server_key: "7618b31b8ef3ac326ce5d1d302047625"
server_key_iv: "209b4b94a13989f418129f23c97b1c25"
server_key_version: "encryptionkey@0"
I20260926 20:52:47.508927 15410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:47.535449 15410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:47.539634 15410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:47.539920 15410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:47.540851 15410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:47.542147 15410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:47.542213 15410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:47.542342 15410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:47.542399 15410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:47.574862 15410 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:33315
I20260926 20:52:47.574923 15537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:33315 every 8 connection(s)
I20260926 20:52:47.576120 15410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:52:47.581816 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 15410
I20260926 20:52:47.582012 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:52:47.582682 9190 external_mini_cluster.cc:1428] Setting key 5c329931a4d9861846cffbf9282e5c0f
Sep 26 20:52:47 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455967, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:52:47.597054 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:47.578608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:56075 (local address 127.8.249.190:44135)
0926 20:52:47.579085 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:47.579089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:47.580098 (+ 1009us) server_negotiation.cc:407] Connection header received
0926 20:52:47.582115 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:47.582121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:47.582215 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:47.582334 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:47.584046 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:47.584909 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:47.586513 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:47.586959 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:47.587126 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:47.590478 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:47.590483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:47.590486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:47.590517 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:47.593943 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:47.594594 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:47.594602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:47.594608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:47.594754 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:47.595161 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:47.595167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:47.595171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:47.595553 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:47.595704 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:47.596318 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 20:52:47.596579 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":271,"threads_started":1}
I20260926 20:52:47.597641 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:47.579163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.129:56075)
0926 20:52:47.579947 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:47.579984 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:47.581792 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:47.582583 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:47.582591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:47.583138 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:47.583878 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:47.583897 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:47.585075 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:47.585079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:47.586336 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:47.586347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:47.587196 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:47.587205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:47.587445 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:47.588132 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:47.588154 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:47.590257 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:47.594068 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:47.594074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:47.594086 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:47.594480 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:47.594876 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:47.594879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:47.594881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:47.595049 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:47.595673 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:47.595678 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:47.595957 (+ 279us) client_negotiation.cc:769] Sending connection context
0926 20:52:47.596174 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 20:52:47.596523 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":251,"threads_started":1}
I20260926 20:52:47.599359 15538 heartbeater.cc:347] Connected to a master server at 127.8.249.190:44135
I20260926 20:52:47.599663 15538 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:47.600558 15538 heartbeater.cc:511] Master 127.8.249.190:44135 requested a full tablet report, sending...
I20260926 20:52:47.603021 15349 ts_manager.cc:194] Registered new tserver with Master: db22faab4b5d414fadbce62b9eec0d9e (127.8.249.129:33315)
WARNING: no policy specified for oryx/127.8.249.130@KRBTEST.COM; defaulting to no policy
I20260926 20:52:47.604597 15349 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:56075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab.
Entry for principal oryx/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab.
I20260926 20:52:47.630550 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44135
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab
--principal=oryx/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:47.951364 15544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:47.951713 15544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:47.951882 15544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:47.959343 15544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:47.959492 15544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:47.959678 15544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:52:47.969601 15544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab
--principal=oryx/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:47.970979 15544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:47.972285 15544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:47.985455 15552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:47.985502 15549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:47.985529 15550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:47.986546 15544 server_base.cc:1034] running on GCE node
I20260926 20:52:47.987174 15544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:47.988046 15544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:47.989241 15544 hybrid_clock.cc:648] HybridClock initialized: now 1790455967989201 us; error 32 us; skew 500 ppm
I20260926 20:52:47.989560 15544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:47 dist-test-slave-2dg3 krb5kdc[15304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455967, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:47.992862 15544 init.cc:377] Logged in from keytab as oryx/127.8.249.130@KRBTEST.COM (short username oryx)
I20260926 20:52:47.994272 15544 webserver.cc:460] Webserver started at http://127.8.249.130:45447/ using document root <none> and password file <none>
I20260926 20:52:47.995009 15544 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:47.995147 15544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:47.995445 15544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:47.998214 15544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "00012a67b7e844bd8563f8cfaa0013b1"
format_stamp: "Formatted at 2026-09-26 20:52:47 on dist-test-slave-2dg3"
server_key: "4c3d0eae8311327d4f910187c84316e3"
server_key_iv: "bc1d0d49ea647769aa0ee74c6728d16c"
server_key_version: "encryptionkey@0"
I20260926 20:52:47.999030 15544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "00012a67b7e844bd8563f8cfaa0013b1"
format_stamp: "Formatted at 2026-09-26 20:52:47 on dist-test-slave-2dg3"
server_key: "4c3d0eae8311327d4f910187c84316e3"
server_key_iv: "bc1d0d49ea647769aa0ee74c6728d16c"
server_key_version: "encryptionkey@0"
I20260926 20:52:48.004333 15544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:52:48.008028 15559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:48.009349 15544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:52:48.009542 15544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "00012a67b7e844bd8563f8cfaa0013b1"
format_stamp: "Formatted at 2026-09-26 20:52:47 on dist-test-slave-2dg3"
server_key: "4c3d0eae8311327d4f910187c84316e3"
server_key_iv: "bc1d0d49ea647769aa0ee74c6728d16c"
server_key_version: "encryptionkey@0"
I20260926 20:52:48.009743 15544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:48.021431 15544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:48.025053 15544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:48.025352 15544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:48.026265 15544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:48.027638 15544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:48.027737 15544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:48.027886 15544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:48.027958 15544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:48.061048 15544 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:42309
I20260926 20:52:48.061088 15671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:42309 every 8 connection(s)
I20260926 20:52:48.062409 15544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:52:48.070911 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 15544
I20260926 20:52:48.071079 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:52:48.071525 9190 external_mini_cluster.cc:1428] Setting key 66172484a93b185765bb2bade2693cc9
Sep 26 20:52:48 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455967, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:52:48.083045 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:48.064949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:54265 (local address 127.8.249.190:44135)
0926 20:52:48.065134 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:48.065138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:48.066258 (+ 1120us) server_negotiation.cc:407] Connection header received
0926 20:52:48.068058 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:48.068063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:48.068153 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:48.068261 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:48.069654 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:48.070350 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:48.072141 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:48.072599 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:48.072702 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:48.076873 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:48.076879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:48.076882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:48.076914 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:48.079616 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:48.080455 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:48.080461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:48.080465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:48.080639 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:48.081077 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:48.081085 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:48.081090 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:48.081487 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:48.081654 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:48.082307 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 20:52:48.082572 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:52:48.083753 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:48.065438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.130:54265)
0926 20:52:48.066106 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:48.066143 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:48.067800 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:48.068480 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:48.068488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:48.068958 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:48.069500 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:48.069516 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:48.070513 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:48.070521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:48.071964 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:48.071995 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:48.072813 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:48.072820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:48.073105 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:48.073887 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:48.073909 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:48.076643 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:48.079751 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:48.079763 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:48.079785 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:48.080301 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:48.080758 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:48.080761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:48.080763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:48.080959 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:48.081609 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:48.081615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:48.081941 (+ 326us) client_negotiation.cc:769] Sending connection context
0926 20:52:48.082166 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 20:52:48.082554 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":263,"threads_started":1}
I20260926 20:52:48.085322 15672 heartbeater.cc:347] Connected to a master server at 127.8.249.190:44135
I20260926 20:52:48.085657 15672 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:48.086493 15672 heartbeater.cc:511] Master 127.8.249.190:44135 requested a full tablet report, sending...
I20260926 20:52:48.088088 15349 ts_manager.cc:194] Registered new tserver with Master: 00012a67b7e844bd8563f8cfaa0013b1 (127.8.249.130:42309)
I20260926 20:52:48.088871 15349 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:54265
WARNING: no policy specified for oryx/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab.
Entry for principal oryx/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab.
I20260926 20:52:48.122184 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44135
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab
--principal=oryx/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:48.440589 15677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:48.440930 15677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:48.441097 15677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:48.448294 15677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:48.448474 15677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:48.448670 15677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:52:48.458732 15677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab
--principal=oryx/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:48.460148 15677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:48.461565 15677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:48.474643 15682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:48.474615 15685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:48.474685 15683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:48.475776 15677 server_base.cc:1034] running on GCE node
I20260926 20:52:48.476356 15677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:48.477279 15677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:48.478520 15677 hybrid_clock.cc:648] HybridClock initialized: now 1790455968478451 us; error 55 us; skew 500 ppm
I20260926 20:52:48.478973 15677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:48 dist-test-slave-2dg3 krb5kdc[15304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455968, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:48.482220 15677 init.cc:377] Logged in from keytab as oryx/127.8.249.131@KRBTEST.COM (short username oryx)
I20260926 20:52:48.483707 15677 webserver.cc:460] Webserver started at http://127.8.249.131:40765/ using document root <none> and password file <none>
I20260926 20:52:48.484341 15677 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:48.484442 15677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:48.484678 15677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:48.487643 15677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "f21ee59a5d9f4b7f867c0f978165a969"
format_stamp: "Formatted at 2026-09-26 20:52:48 on dist-test-slave-2dg3"
server_key: "d008e7529d38e179b17f8b280db7fbb8"
server_key_iv: "9a4c4ec8cdc6c072f7b464095bdd3616"
server_key_version: "encryptionkey@0"
I20260926 20:52:48.488375 15677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "f21ee59a5d9f4b7f867c0f978165a969"
format_stamp: "Formatted at 2026-09-26 20:52:48 on dist-test-slave-2dg3"
server_key: "d008e7529d38e179b17f8b280db7fbb8"
server_key_iv: "9a4c4ec8cdc6c072f7b464095bdd3616"
server_key_version: "encryptionkey@0"
I20260926 20:52:48.493469 15677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:52:48.496796 15692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:48.498208 15677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 20:52:48.498401 15677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "f21ee59a5d9f4b7f867c0f978165a969"
format_stamp: "Formatted at 2026-09-26 20:52:48 on dist-test-slave-2dg3"
server_key: "d008e7529d38e179b17f8b280db7fbb8"
server_key_iv: "9a4c4ec8cdc6c072f7b464095bdd3616"
server_key_version: "encryptionkey@0"
I20260926 20:52:48.498615 15677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:48.515847 15677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:48.519747 15677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:48.520040 15677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:48.521035 15677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:48.522396 15677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:48.522470 15677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:48.522662 15677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:48.522711 15677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:48.555469 15677 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:39939
I20260926 20:52:48.555506 15804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:39939 every 8 connection(s)
I20260926 20:52:48.556798 15677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:52:48.562134 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 15677
I20260926 20:52:48.562310 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:52:48.562885 9190 external_mini_cluster.cc:1428] Setting key fa22cd78b712cb539b55a102279dd192
Sep 26 20:52:48 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455968, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:52:48.577138 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:48.559348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:40003 (local address 127.8.249.190:44135)
0926 20:52:48.559515 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:48.559520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:48.560759 (+ 1239us) server_negotiation.cc:407] Connection header received
0926 20:52:48.563074 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:48.563079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:48.563149 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:48.563241 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:48.565125 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:48.566174 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:48.567711 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:48.568094 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:48.568188 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:48.571744 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:48.571749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:48.571752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:48.571789 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:48.574150 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:48.574906 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:48.574909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:48.574918 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:48.575044 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:48.575490 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:48.575499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:48.575501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:48.575756 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:48.575921 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:48.576560 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 20:52:48.576779 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:52:48.578454 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:48.559893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.131:40003)
0926 20:52:48.560608 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:48.560644 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:48.562675 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:48.563539 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:48.563549 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:48.564081 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:48.564938 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:48.564964 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:48.566312 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:48.566316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:48.567583 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:48.567594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:48.568266 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:48.568270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:48.568465 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:48.569129 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:48.569154 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:48.571535 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:48.574287 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:48.574296 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:48.574312 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:48.574734 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:48.575167 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:48.575173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:48.575177 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:48.575368 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:48.575881 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:48.575888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:48.576207 (+ 319us) client_negotiation.cc:769] Sending connection context
0926 20:52:48.576473 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 20:52:48.576970 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":274,"threads_started":1}
I20260926 20:52:48.579839 15805 heartbeater.cc:347] Connected to a master server at 127.8.249.190:44135
I20260926 20:52:48.580159 15805 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:48.580907 15805 heartbeater.cc:511] Master 127.8.249.190:44135 requested a full tablet report, sending...
I20260926 20:52:48.582154 15349 ts_manager.cc:194] Registered new tserver with Master: f21ee59a5d9f4b7f867c0f978165a969 (127.8.249.131:39939)
I20260926 20:52:48.583006 15349 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:40003
I20260926 20:52:48.590418 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:52:48.607890 15538 heartbeater.cc:503] Master 127.8.249.190:44135 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.8.249.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.249.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.249.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.189.keytab.
Entry for principal oryx/127.8.249.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.189.keytab.
I20260926 20:52:48.639823 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.189:46689
--webserver_interface=127.8.249.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.189.keytab
--principal=oryx/127.8.249.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:44135,127.8.249.189:46689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:48.982560 15810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:48.982939 15810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:48.983135 15810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:48.990648 15810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:48.990805 15810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:48.990949 15810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:48.991055 15810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:48.991128 15810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:49.001592 15810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:44135,127.8.249.189:46689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.189.keytab
--principal=oryx/127.8.249.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.189:46689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.249.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
W20260926 20:52:49.002013 15810 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.249.190:44135,127.8.249.189:46689'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 20:52:49.003173 15810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:49.004465 15810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:49.016426 15815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:49.016501 15816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:49.016521 15818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:49.017515 15810 server_base.cc:1034] running on GCE node
I20260926 20:52:49.018302 15810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:49.019659 15810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:49.020905 15810 hybrid_clock.cc:648] HybridClock initialized: now 1790455969020827 us; error 55 us; skew 500 ppm
I20260926 20:52:49.021245 15810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:49 dist-test-slave-2dg3 krb5kdc[15304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455969, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:49.024348 15810 init.cc:377] Logged in from keytab as oryx/127.8.249.189@KRBTEST.COM (short username oryx)
I20260926 20:52:49.025732 15810 webserver.cc:460] Webserver started at http://127.8.249.189:35413/ using document root <none> and password file <none>
I20260926 20:52:49.026363 15810 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:49.026448 15810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:49.026695 15810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:49.029595 15810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/data/instance:
uuid: "aefc04b1a7444d8fbe73b7a2388d47d1"
format_stamp: "Formatted at 2026-09-26 20:52:49 on dist-test-slave-2dg3"
server_key: "a774c9900c2396cdf0688794540fc40c"
server_key_iv: "022868f03e64a71d57f88639bc3f0f22"
server_key_version: "encryptionkey@0"
I20260926 20:52:49.030301 15810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/wal/instance:
uuid: "aefc04b1a7444d8fbe73b7a2388d47d1"
format_stamp: "Formatted at 2026-09-26 20:52:49 on dist-test-slave-2dg3"
server_key: "a774c9900c2396cdf0688794540fc40c"
server_key_iv: "022868f03e64a71d57f88639bc3f0f22"
server_key_version: "encryptionkey@0"
I20260926 20:52:49.035409 15810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:52:49.038785 15825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:49.040398 15810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:52:49.040586 15810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/wal
uuid: "aefc04b1a7444d8fbe73b7a2388d47d1"
format_stamp: "Formatted at 2026-09-26 20:52:49 on dist-test-slave-2dg3"
server_key: "a774c9900c2396cdf0688794540fc40c"
server_key_iv: "022868f03e64a71d57f88639bc3f0f22"
server_key_version: "encryptionkey@0"
I20260926 20:52:49.040812 15810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:49.067839 15810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:49.072067 15810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:49.072355 15810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:49.091583 15810 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.189:46689
I20260926 20:52:49.091602 15876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.189:46689 every 8 connection(s)
I20260926 20:52:49.092239 15672 heartbeater.cc:503] Master 127.8.249.190:44135 was elected leader, sending a full tablet report...
I20260926 20:52:49.093348 15810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/data/info.pb
I20260926 20:52:49.097137 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:49.100533 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 15810
I20260926 20:52:49.105237 15877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:52:49 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455969, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.189@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
Sep 26 20:52:49 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.249.189@KRBTEST.COM
I20260926 20:52:49.127803 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.108495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.8.249.190:44135)
0926 20:52:49.108987 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.108994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.109446 (+ 452us) server_negotiation.cc:407] Connection header received
0926 20:52:49.109646 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.109651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.109722 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.109836 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.111200 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.112210 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.114005 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.114380 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.114491 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.121129 (+ 6638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.121135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.121139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.121178 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.124579 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.125422 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.125427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.125435 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.125594 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.125989 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.125996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.125999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.126307 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.126484 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.127213 (+ 729us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.127440 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":240,"threads_started":1}
I20260926 20:52:49.128911 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.102947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.8.249.189:46689)
0926 20:52:49.103673 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.103688 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.103734 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:52:49.105825 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.105858 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.106389 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.106841 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.108213 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.110063 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.111645 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.112631 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.112757 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.122003 (+ 9246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.122016 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.122032 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.122065 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.125445 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.126044 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.126052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.126055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.126178 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.126539 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.126544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.126547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.126947 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.127246 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.127577 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.127821 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":510,"thread_start_us":308,"threads_started":1}
I20260926 20:52:49.128940 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.108597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:38492)
0926 20:52:49.109294 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.109336 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.109521 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.109893 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.109900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.110714 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:49.111055 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.111067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.112334 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.112337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.113870 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.113881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.114602 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.114611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.114739 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.118133 (+ 3394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:49.118163 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:49.120892 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:49.124715 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.124723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.124732 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.125301 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.125704 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.125709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.125719 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.125873 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.126451 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:49.126461 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:49.126847 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.127132 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.127635 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":44,"thread_start_us":347,"threads_started":1}
I20260926 20:52:49.131047 15877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:52:49 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455969, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.189@KRBTEST.COM for oryx/127.8.249.189@KRBTEST.COM
I20260926 20:52:49.142879 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.131705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46689 (local address 127.0.0.1:45128)
0926 20:52:49.131880 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.131917 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.132021 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.132312 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.132316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.132691 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:49.132808 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.132815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.133776 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.133781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.134618 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.134628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.135319 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.135324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.135544 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.136042 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:49.136058 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:49.138041 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:49.140909 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.140913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.140916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.141273 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.141562 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.141565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.141567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.141686 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.142111 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:49.142114 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:49.142222 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.142327 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.142534 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":3}
I20260926 20:52:49.142956 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.131832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.8.249.189:46689)
0926 20:52:49.131980 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.131985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.132001 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:52:49.132138 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.132143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.132205 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.132298 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.132938 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.133642 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.134749 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.135179 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.135309 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.138220 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.138225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.138229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.138258 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.140790 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.141366 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.141369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.141372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.141475 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.141769 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.141772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.141774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.142022 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.142131 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.142405 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.142635 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":47}
I20260926 20:52:49.147786 15877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: Bootstrap starting.
I20260926 20:52:49.150709 15877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:49.151744 15877 log.cc:824] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:49.154265 15877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: No bootstrap required, opened a new log
I20260926 20:52:49.160928 15877 raft_consensus.cc:348] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } }
I20260926 20:52:49.161207 15877 raft_consensus.cc:374] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:49.161264 15877 raft_consensus.cc:729] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefc04b1a7444d8fbe73b7a2388d47d1, State: Initialized, Role: FOLLOWER
I20260926 20:52:49.161741 15877 consensus_queue.cc:260] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } }
I20260926 20:52:49.162528 15884 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } } }
I20260926 20:52:49.162846 15884 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:52:49.163431 15877 sys_catalog.cc:564] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:52:49.173276 15895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:52:49.173424 15895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:52:49.174651 15810 master_runner.cc:428] Detected that this master aefc04b1a7444d8fbe73b7a2388d47d1 is joining an existing cluster
I20260926 20:52:49.174732 15810 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.249.189:46689
I20260926 20:52:49.206640 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.179171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.8.249.190:44135)
0926 20:52:49.179393 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.179401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.189950 (+ 10549us) server_negotiation.cc:407] Connection header received
0926 20:52:49.190595 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.190602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.190719 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.190795 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.192328 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.193432 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.195490 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.196232 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.196374 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.199724 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.199732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.199738 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.199782 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.203437 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.204232 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.204238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.204242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.204387 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.204847 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.204852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.204856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.205211 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.205349 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.205960 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.206220 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 20:52:49.208179 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.178746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:38502)
0926 20:52:49.189828 (+ 11082us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.189855 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.190396 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.191036 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.191041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.191740 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:49.192160 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.192174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.193604 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.193610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.195314 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.195327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.196518 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.196529 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.196703 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.198311 (+ 1608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:49.198332 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:49.199504 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:49.203587 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.203592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.203597 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.204087 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.204526 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.204531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.204535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.204710 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.205432 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:49.205442 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:49.205659 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.207306 (+ 1647us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.207690 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10913,"thread_start_us":22952,"threads_started":1}
I20260926 20:52:49.213896 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.178705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.8.249.189:46689)
0926 20:52:49.178978 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.178986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.196001 (+ 17015us) server_negotiation.cc:407] Connection header received
0926 20:52:49.196206 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.196211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.196271 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.196338 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.198215 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.199430 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.202145 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.202989 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.203108 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.205718 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.205727 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.205735 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.205773 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.209970 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.210922 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.210929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.210934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.211105 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.211857 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.211869 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.211872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.212554 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.212690 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.213246 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.213534 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260926 20:52:49.215149 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.178853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46689 (local address 127.0.0.1:45140)
0926 20:52:49.195910 (+ 17057us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.195929 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.196092 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.197088 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.197092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.197749 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:49.198034 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.198047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.199635 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.199643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.201996 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.202012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.203453 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.203461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.203660 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.204333 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:49.204360 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:49.205520 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:49.210165 (+ 4645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.210171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.210176 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.210772 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.211257 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.211262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.211266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.211444 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.212768 (+ 1324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:49.212774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:49.212938 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.214135 (+ 1197us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.214552 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16903,"thread_start_us":11124,"threads_started":1}
WARNING: no policy specified for oryx/127.8.249.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.249.188@KRBTEST.COM" created.
Sep 26 20:52:49 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455966, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.190@KRBTEST.COM for oryx/127.8.249.189@KRBTEST.COM
I20260926 20:52:49.250042 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.219638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.8.249.189:46689)
0926 20:52:49.220708 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.220716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.231641 (+ 10925us) server_negotiation.cc:407] Connection header received
0926 20:52:49.232102 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.232108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.232244 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.232337 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.234200 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.235430 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.237412 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.238106 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.238204 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.242595 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.242604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.242615 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.242660 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.246998 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.247872 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.247879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.247882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.248042 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.248488 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.248495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.248500 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.248908 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.249056 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.249587 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.249742 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914}
I20260926 20:52:49.250272 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.217380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46689 (local address 127.0.0.1:45152)
0926 20:52:49.231433 (+ 14053us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.231491 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.231834 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.232665 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.232672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.233681 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:49.234018 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.234033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.235619 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.235625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.237244 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.237260 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.238483 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.238489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.238706 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.239449 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:49.239473 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:49.242370 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:49.247141 (+ 4771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.247153 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.247161 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.247734 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.248170 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.248175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.248179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.248364 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.249050 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:49.249060 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:49.249271 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.249493 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.249850 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13706,"thread_start_us":13555,"threads_started":1}
I20260926 20:52:49.251149 15884 raft_consensus.cc:482] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:49.251416 15884 raft_consensus.cc:504] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } }
I20260926 20:52:49.252763 15884 leader_election.cc:288] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135)
I20260926 20:52:49.253146 15349 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.249.189:46689: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1"
member_type: NON_VOTER
last_known_addr {
host: "127.8.249.189"
port: 46689
}
attrs {
promote: true
}
}
dest_uuid: "9d6c916c8ec441e6be3173bb0f696d38"
cas_config_opid_index: -1
I20260926 20:52:49.253592 15359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d6c916c8ec441e6be3173bb0f696d38" is_pre_election: true
I20260926 20:52:49.254315 15349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } }
I20260926 20:52:49.254716 15829 leader_election.cc:302] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: aefc04b1a7444d8fbe73b7a2388d47d1; no voters: 9d6c916c8ec441e6be3173bb0f696d38
I20260926 20:52:49.255260 15884 raft_consensus.cc:2726] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.249.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.188.keytab.
Entry for principal oryx/127.8.249.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.188.keytab.
I20260926 20:52:49.257345 15908 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER aefc04b1a7444d8fbe73b7a2388d47d1 (127.8.249.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } } }
I20260926 20:52:49.258587 15852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:49.258920 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.188:35777
--webserver_interface=127.8.249.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.188.keytab
--principal=oryx/127.8.249.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:44135,127.8.249.189:46689,127.8.249.188:35777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:52:49.262518 15852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 FOLLOWER]: Refusing update from remote peer 9d6c916c8ec441e6be3173bb0f696d38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 20:52:49.272915 15910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:49.286185 15908 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.249.189:46689
I20260926 20:52:49.287029 15810 master.cc:537] Master@127.8.249.189:46689 shutting down...
I20260926 20:52:49.288619 15907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } } }
I20260926 20:52:49.289007 15907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:49.336395 15852 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER aefc04b1a7444d8fbe73b7a2388d47d1 (127.8.249.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } } }
I20260926 20:52:49.340838 15918 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } } }
I20260926 20:52:49.341204 15918 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:52:49.341324 15884 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d6c916c8ec441e6be3173bb0f696d38. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } } }
I20260926 20:52:49.341549 15884 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:52:49.341583 15915 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } } }
I20260926 20:52:49.341908 15915 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:52:49.344358 15810 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 LEARNER]: Raft consensus shutting down.
I20260926 20:52:49.344821 15810 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 LEARNER]: Raft consensus is shut down!
I20260926 20:52:49.344977 15810 tablet_replica.cc:330] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: stopping tablet replica
I20260926 20:52:49.351882 15810 master.cc:552] Master@127.8.249.189:46689 shutdown complete.
I20260926 20:52:49.351979 15810 master_runner.cc:305] Clearing existing system tablet
I20260926 20:52:49.352414 15810 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:49.352507 15810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:49.355587 15810 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260926 20:52:49.357270 15920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:49.357631 15810 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:49.357904 15810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 20:52:49.357991 15810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/wal
uuid: "aefc04b1a7444d8fbe73b7a2388d47d1"
format_stamp: "Formatted at 2026-09-26 20:52:49 on dist-test-slave-2dg3"
server_key: "a774c9900c2396cdf0688794540fc40c"
server_key_iv: "022868f03e64a71d57f88639bc3f0f22"
server_key_version: "encryptionkey@0"
I20260926 20:52:49.362669 15810 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:49.370433 15810 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:52:49.370646 15810 log.cc:1196] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 20:52:49.371161 15810 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: Deleting consensus metadata
I20260926 20:52:49.371420 15810 master_runner.cc:315] Copying system tablet from 127.8.249.190:44135
I20260926 20:52:49.375988 15810 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: tablet copy: Beginning tablet copy session from remote peer at address 127.8.249.190:44135
I20260926 20:52:49.392238 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.376808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.8.249.190:44135)
0926 20:52:49.376995 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.377001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.377555 (+ 554us) server_negotiation.cc:407] Connection header received
0926 20:52:49.377785 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.377791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.377876 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.377987 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.379117 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.380257 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.382378 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.383113 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.383212 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.385719 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.385727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.385732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.385771 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.389241 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.390078 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.390085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.390089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.390237 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.390630 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.390635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.390638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.390979 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.391131 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.391616 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.391825 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:52:49.393267 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.376749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:38518)
0926 20:52:49.377418 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.377444 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.377600 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.378053 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.378058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.378665 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:49.378948 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.378961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.380419 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.380426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.382183 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.382196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.383472 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.383478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.383644 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.384358 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:49.384379 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:49.385515 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:49.389395 (+ 3880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.389403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.389409 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.389913 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.390363 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.390368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.390371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.390521 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.391104 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:49.391109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:49.391299 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.392502 (+ 1203us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.392813 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":401,"threads_started":1}
I20260926 20:52:49.394105 15369 tablet_copy_service.cc:140] P 9d6c916c8ec441e6be3173bb0f696d38: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer aefc04b1a7444d8fbe73b7a2388d47d1 ({username='oryx', principal='oryx/127.8.249.189@KRBTEST.COM'} at 127.0.0.1:38518)
I20260926 20:52:49.394342 15369 tablet_copy_service.cc:161] P 9d6c916c8ec441e6be3173bb0f696d38: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer aefc04b1a7444d8fbe73b7a2388d47d1 at {username='oryx', principal='oryx/127.8.249.189@KRBTEST.COM'} at 127.0.0.1:38518: session id = aefc04b1a7444d8fbe73b7a2388d47d1-00000000000000000000000000000000
I20260926 20:52:49.398702 15369 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:52:49.400804 15810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:52:49.407757 15810 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: tablet copy: Starting download of 0 data blocks...
I20260926 20:52:49.408128 15810 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: tablet copy: Starting download of 1 WAL segments...
I20260926 20:52:49.410760 15810 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:52:49.414995 15369 tablet_copy_service.cc:342] P 9d6c916c8ec441e6be3173bb0f696d38: Request end of tablet copy session aefc04b1a7444d8fbe73b7a2388d47d1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.249.189@KRBTEST.COM'} at 127.0.0.1:38518
I20260926 20:52:49.415169 15369 tablet_copy_service.cc:433] P 9d6c916c8ec441e6be3173bb0f696d38: ending tablet copy session aefc04b1a7444d8fbe73b7a2388d47d1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer aefc04b1a7444d8fbe73b7a2388d47d1
I20260926 20:52:49.425542 15810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:49.428046 15927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:49.432389 15928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:49.440114 15810 server_base.cc:1034] running on GCE node
W20260926 20:52:49.440022 15930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:49.440690 15810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:49.441380 15810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:49.442621 15810 hybrid_clock.cc:648] HybridClock initialized: now 1790455969442546 us; error 33 us; skew 500 ppm
I20260926 20:52:49.442953 15810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:49 dist-test-slave-2dg3 krb5kdc[15304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455969, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:49.445194 15810 init.cc:377] Logged in from keytab as oryx/127.8.249.189@KRBTEST.COM (short username oryx)
I20260926 20:52:49.446481 15810 webserver.cc:460] Webserver started at http://127.8.249.189:33349/ using document root <none> and password file <none>
I20260926 20:52:49.446830 15810 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:49.446902 15810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:49.449800 15810 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:52:49.451754 15937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:49.452265 15810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:52:49.452463 15810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/wal
uuid: "aefc04b1a7444d8fbe73b7a2388d47d1"
format_stamp: "Formatted at 2026-09-26 20:52:49 on dist-test-slave-2dg3"
server_key: "a774c9900c2396cdf0688794540fc40c"
server_key_iv: "022868f03e64a71d57f88639bc3f0f22"
server_key_version: "encryptionkey@0"
I20260926 20:52:49.452636 15810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:49.475998 15810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:49.480398 15810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:49.523252 15810 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.189:46689
I20260926 20:52:49.523274 15988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.189:46689 every 8 connection(s)
I20260926 20:52:49.524422 15810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-1/data/info.pb
I20260926 20:52:49.533310 15989 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:52:49.536326 15989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: Bootstrap starting.
I20260926 20:52:49.546551 15989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:52:49.547058 15989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: Bootstrap complete.
I20260926 20:52:49.547909 15989 raft_consensus.cc:348] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } }
I20260926 20:52:49.548087 15989 raft_consensus.cc:729] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aefc04b1a7444d8fbe73b7a2388d47d1, State: Initialized, Role: LEARNER
I20260926 20:52:49.548332 15989 consensus_queue.cc:260] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } }
I20260926 20:52:49.549126 15992 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } } }
I20260926 20:52:49.549453 15989 sys_catalog.cc:564] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:49.549485 15992 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:52:49.574719 16003 catalog_manager.cc:1259] Loaded cluster ID: 986a280ad246442aafa35051eae5c0c7
I20260926 20:52:49.574847 16003 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: loading cluster ID for follower catalog manager: success
I20260926 20:52:49.578264 16003 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: acquiring CA information for follower catalog manager: success
I20260926 20:52:49.580519 16003 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:52:49.585412 15805 heartbeater.cc:503] Master 127.8.249.190:44135 was elected leader, sending a full tablet report...
I20260926 20:52:49.662976 15908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } } }
I20260926 20:52:49.663273 15908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:49.670182 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.663253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46689 (local address 127.0.0.1:45168)
0926 20:52:49.663578 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.663602 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.663738 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.664780 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.664785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.664817 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:49.665023 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.665033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.666680 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.666686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.668058 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.668072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.669311 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.669317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.669543 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.669610 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.669732 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.669881 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260926 20:52:49.670527 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.663703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.8.249.189:46689)
0926 20:52:49.664443 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.664447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.664464 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:52:49.664523 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.664527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.664593 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.664739 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:49.665178 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.666527 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.668244 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.669139 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.669283 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.669559 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.669759 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.669904 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":484,"threads_started":1}
I20260926 20:52:49.671533 15992 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d6c916c8ec441e6be3173bb0f696d38. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } } }
I20260926 20:52:49.671814 15992 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:52:49.672482 15908 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38: attempting to promote NON_VOTER aefc04b1a7444d8fbe73b7a2388d47d1 to VOTER
I20260926 20:52:49.673390 15908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } }
I20260926 20:52:49.675082 15964 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 LEARNER]: Refusing update from remote peer 9d6c916c8ec441e6be3173bb0f696d38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 20:52:49.675650 15907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 20:52:49.678679 15908 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aefc04b1a7444d8fbe73b7a2388d47d1 (127.8.249.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } }
I20260926 20:52:49.681540 15908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } }
I20260926 20:52:49.681787 15908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:49.681710 15907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } }
I20260926 20:52:49.681900 15907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:49.682622 15964 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aefc04b1a7444d8fbe73b7a2388d47d1 (127.8.249.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } }
I20260926 20:52:49.686124 15992 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } }
I20260926 20:52:49.686378 15992 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:52:49.712018 15911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:49.712484 15911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:49.712641 15911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:49.724632 15911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:49.724785 15911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:49.724848 15911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:49.724922 15911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:49.724980 15911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:49.741153 15911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:44135,127.8.249.189:46689,127.8.249.188:35777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.188.keytab
--principal=oryx/127.8.249.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.188:35777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.249.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:49.742851 15911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:49.744517 15911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:49.757884 16010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:49.757918 16011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:49.758692 16013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:49.759199 15911 server_base.cc:1034] running on GCE node
I20260926 20:52:49.759963 15911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:49.761489 15911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:49.762769 15911 hybrid_clock.cc:648] HybridClock initialized: now 1790455969762729 us; error 62 us; skew 500 ppm
I20260926 20:52:49.763203 15911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:49 dist-test-slave-2dg3 krb5kdc[15304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455969, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:49.766422 15911 init.cc:377] Logged in from keytab as oryx/127.8.249.188@KRBTEST.COM (short username oryx)
I20260926 20:52:49.768010 15911 webserver.cc:460] Webserver started at http://127.8.249.188:46851/ using document root <none> and password file <none>
I20260926 20:52:49.768747 15911 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:49.768879 15911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:49.769186 15911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:49.772271 15911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/data/instance:
uuid: "0a04345301c847df90874e0f9cc23c84"
format_stamp: "Formatted at 2026-09-26 20:52:49 on dist-test-slave-2dg3"
server_key: "89acb5b7cc970a0ac422b0078be972ea"
server_key_iv: "4ad6ec9d67cc9df1f0d84629048fff86"
server_key_version: "encryptionkey@0"
I20260926 20:52:49.773079 15911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/wal/instance:
uuid: "0a04345301c847df90874e0f9cc23c84"
format_stamp: "Formatted at 2026-09-26 20:52:49 on dist-test-slave-2dg3"
server_key: "89acb5b7cc970a0ac422b0078be972ea"
server_key_iv: "4ad6ec9d67cc9df1f0d84629048fff86"
server_key_version: "encryptionkey@0"
I20260926 20:52:49.778575 15911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:52:49.782446 16020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:49.784184 15911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:49.784399 15911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/wal
uuid: "0a04345301c847df90874e0f9cc23c84"
format_stamp: "Formatted at 2026-09-26 20:52:49 on dist-test-slave-2dg3"
server_key: "89acb5b7cc970a0ac422b0078be972ea"
server_key_iv: "4ad6ec9d67cc9df1f0d84629048fff86"
server_key_version: "encryptionkey@0"
I20260926 20:52:49.784622 15911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:49.814036 15911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:49.818560 15911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:49.818877 15911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:49.836549 15911 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.188:35777
I20260926 20:52:49.836575 16071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.188:35777 every 8 connection(s)
I20260926 20:52:49.837937 15911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/data/info.pb
I20260926 20:52:49.839627 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 15911
I20260926 20:52:49.842518 16072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:49.849531 16072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:52:49 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.249.188@KRBTEST.COM
Sep 26 20:52:49 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455969, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.188@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:52:49.862628 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.841662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.8.249.188:35777)
0926 20:52:49.842419 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.842431 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.842471 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:52:49.844140 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.844163 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.844548 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.844973 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.846443 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.847894 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.849130 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.849973 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.850172 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.855245 (+ 5073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.855252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.855262 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.855293 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.858161 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.859007 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.859013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.859020 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.859185 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.859579 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.859581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.859583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.860038 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.860350 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.860798 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.861207 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"spinlock_wait_cycles":13056,"thread_start_us":352,"threads_started":1}
I20260926 20:52:49.867100 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.852298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.8.249.190:44135)
0926 20:52:49.852503 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.852508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.853231 (+ 723us) server_negotiation.cc:407] Connection header received
0926 20:52:49.853506 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.853512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.853618 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.853735 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.854998 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.855807 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.857245 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.857673 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.857799 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.861191 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.861197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.861203 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.861242 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.864386 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.865175 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.865178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.865181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.865328 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.865757 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.865761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.865763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.866011 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.866122 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.866551 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.866749 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:52:49.867166 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.852317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:38524)
0926 20:52:49.853103 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.853136 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.853310 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.853810 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.853817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.854561 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:49.854841 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.854854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.855937 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.855944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.857108 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.857119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.857860 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.857871 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.858031 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.858594 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:49.858615 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:49.860911 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:49.864518 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.864528 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.864535 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.865068 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.865468 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.865475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.865478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.865640 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.866119 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:49.866125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:49.866258 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.866442 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.866742 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":358,"threads_started":1}
I20260926 20:52:49.868535 16072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:52:49 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455969, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.188@KRBTEST.COM for oryx/127.8.249.189@KRBTEST.COM
I20260926 20:52:49.880602 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.869229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46689 (local address 127.0.0.1:45180)
0926 20:52:49.869404 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.869423 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.869513 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.869771 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.869774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.870051 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:49.870188 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.870195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.871227 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.871230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.872141 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.872151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.872790 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.872794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.872910 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.873487 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:49.873504 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:49.875620 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:49.878215 (+ 2595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.878220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.878224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.878581 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.879015 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.879021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.879032 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.879164 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.879664 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:49.879669 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:49.879811 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.879945 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.880252 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 20:52:49.880612 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.869245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.8.249.189:46689)
0926 20:52:49.869407 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.869411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.869470 (+ 59us) server_negotiation.cc:407] Connection header received
0926 20:52:49.869605 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.869608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.869665 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.869743 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.870294 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.871095 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.872264 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.872670 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.872762 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.875801 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.875806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.875809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.875839 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.878067 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.878695 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.878702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.878706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.878907 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.879267 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.879274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.879278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.879543 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.879687 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.880031 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.880256 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:52:49.881630 16072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:52:49 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455969, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.188@KRBTEST.COM for oryx/127.8.249.188@KRBTEST.COM
I20260926 20:52:49.893694 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.882163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:35777 (local address 127.0.0.1:46252)
0926 20:52:49.882323 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.882357 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.882451 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.882710 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.882714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.883093 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:49.883222 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.883229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.884133 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.884140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.884999 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.885009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.885724 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.885730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.885921 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.886492 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:49.886510 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:49.888564 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:49.891513 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.891517 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.891520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.891943 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.892294 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.892297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.892299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.892448 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.892875 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:49.892878 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:49.893109 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.893220 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.893403 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 20:52:49.893935 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.882317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.8.249.188:35777)
0926 20:52:49.882444 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.882450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.882466 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:52:49.882547 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.882553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.882605 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.882697 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.883345 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.884021 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.885134 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.885582 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.885710 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.888715 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.888719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.888722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.888749 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.891395 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.892052 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.892056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.892058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.892209 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.892539 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.892541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.892543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.892783 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.892883 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.893349 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.893564 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:52:49.898375 16072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: Bootstrap starting.
I20260926 20:52:49.901568 16072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:49.902513 16072 log.cc:824] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:49.905131 16072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: No bootstrap required, opened a new log
I20260926 20:52:49.912168 16072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } }
I20260926 20:52:49.912520 16072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:49.912655 16072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a04345301c847df90874e0f9cc23c84, State: Initialized, Role: FOLLOWER
I20260926 20:52:49.913316 16072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } }
I20260926 20:52:49.914325 16078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } } }
I20260926 20:52:49.914665 16078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:52:49.915457 16072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:52:49.924738 16089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:52:49.924886 16089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:52:49.928099 15911 master_runner.cc:428] Detected that this master 0a04345301c847df90874e0f9cc23c84 is joining an existing cluster
I20260926 20:52:49.928217 15911 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.249.188:35777
I20260926 20:52:49.954916 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.932369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.8.249.189:46689)
0926 20:52:49.932565 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.932572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.932941 (+ 369us) server_negotiation.cc:407] Connection header received
0926 20:52:49.933265 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.933271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.933358 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.933440 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.934969 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.936028 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.938083 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.938714 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.938864 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.944007 (+ 5143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.944015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.944019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.944079 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.947730 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.951200 (+ 3470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.951208 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.951213 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.951422 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.951975 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.951980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.951983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.952341 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.952500 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.954225 (+ 1725us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.954515 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:52:49.956101 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.932204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46689 (local address 127.0.0.1:45182)
0926 20:52:49.932832 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.932848 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.933103 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.933672 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.933676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.934415 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:49.934780 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.934815 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.936194 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.936199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.937913 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.937927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.939186 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.939192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.939382 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.940115 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:49.940141 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:49.942271 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:49.947912 (+ 5641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.947919 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.947925 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.951017 (+ 3092us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.951601 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.951606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.951612 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.951833 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.952589 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:49.952594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:49.952844 (+ 250us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.955204 (+ 2360us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.955570 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":375,"threads_started":1}
I20260926 20:52:49.957839 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.933115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.8.249.188:35777)
0926 20:52:49.933288 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.933294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.933860 (+ 566us) server_negotiation.cc:407] Connection header received
0926 20:52:49.934082 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.934087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.934147 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.934226 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.942590 (+ 8364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.943710 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.945737 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.946487 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.946603 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.949228 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.949234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.949239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.949271 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.954248 (+ 4977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.955204 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.955208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.955212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.955364 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.955798 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.955803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.955806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.956374 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.956547 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.957245 (+ 698us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.957477 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:52:49.958889 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.933062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:35777 (local address 127.0.0.1:46264)
0926 20:52:49.933771 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.933793 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.933943 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.941054 (+ 7111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.941060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.942034 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:49.942406 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.942422 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.943911 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.943917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.945579 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.945594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.946976 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.946984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.947160 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.947829 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:49.947857 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:49.949034 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:49.954431 (+ 5397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.954436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.954441 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.954940 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.955490 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.955500 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.955504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.955680 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.956768 (+ 1088us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:49.956776 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:49.956974 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.958131 (+ 1157us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.958418 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":370,"threads_started":1}
I20260926 20:52:49.966854 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.932090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38528 (local address 127.8.249.190:44135)
0926 20:52:49.932261 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.932265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.940451 (+ 8186us) server_negotiation.cc:407] Connection header received
0926 20:52:49.940622 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.940628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.940704 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.940786 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.948196 (+ 7410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.949421 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.958078 (+ 8657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.958755 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.958897 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.961174 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.961180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.961183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.961218 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.964029 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.964761 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.964766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.964770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.964920 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.965313 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.965318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.965321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.965610 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.965763 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.966229 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.966458 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:52:49.966949 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.933674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:38528)
0926 20:52:49.940349 (+ 6675us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.940370 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.940501 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.940817 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.940822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.943429 (+ 2607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:49.943775 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.943785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.949674 (+ 5889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.949679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.953937 (+ 4258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.953954 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.959041 (+ 5087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.959048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.959245 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.960015 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:49.960031 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:49.960961 (+ 930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:49.964203 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.964209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.964213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.964619 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.965036 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.965041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.965045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.965190 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.965724 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:49.965729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:49.965929 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.966088 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.966444 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6521,"thread_start_us":6921,"threads_started":1}
I20260926 20:52:49.969914 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 15410
Sep 26 20:52:49 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455966, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.190@KRBTEST.COM for oryx/127.8.249.188@KRBTEST.COM
Sep 26 20:52:49 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.249.129@KRBTEST.COM
I20260926 20:52:49.987746 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.972412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:35777 (local address 127.0.0.1:46274)
0926 20:52:49.972620 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:49.972635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:49.972713 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:49.973427 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:49.973431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:49.973775 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:49.973922 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.973930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.975037 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.975049 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.975950 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:49.975961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.976940 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.976948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:49.977119 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.977833 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:49.977857 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:49.980690 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:49.984583 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.984590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.984594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.985166 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.985666 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:49.985672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:49.985678 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:49.985852 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:49.986568 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:49.986574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:49.986765 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 20:52:49.986980 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 20:52:49.987309 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 20:52:49.988157 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.972455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46274 (local address 127.8.249.188:35777)
0926 20:52:49.972865 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.972869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.972881 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:52:49.972947 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.972950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.973074 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.973171 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.974068 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.974892 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.976127 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.976757 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.976884 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.980912 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.980922 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.980929 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.980976 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.984367 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.985313 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.985319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.985323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.985491 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.985963 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.985969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.985973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.986414 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.986573 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.987363 (+ 790us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.987626 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324}
I20260926 20:52:49.989614 15349 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.249.188:35777: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0a04345301c847df90874e0f9cc23c84"
member_type: NON_VOTER
last_known_addr {
host: "127.8.249.188"
port: 35777
}
attrs {
promote: true
}
}
dest_uuid: "9d6c916c8ec441e6be3173bb0f696d38"
cas_config_opid_index: 6
I20260926 20:52:49.990283 15349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: NON_VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: true } }
I20260926 20:52:49.992520 15964 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 FOLLOWER]: Refusing update from remote peer 9d6c916c8ec441e6be3173bb0f696d38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 20:52:49.993160 15907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 20:52:49.993431 16047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:49.995702 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:49.970972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.8.249.129:33315)
0926 20:52:49.971884 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:49.971894 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:49.971925 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:52:49.971994 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:49.972003 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:49.972302 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:49.974294 (+ 1992us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:49.974560 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.976207 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.977937 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:49.978535 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:49.978688 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:49.984864 (+ 6176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:49.984885 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:49.984908 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:49.984959 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:49.992187 (+ 7228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.993169 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.993182 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.993186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.993342 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:49.993880 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:49.993886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:49.993890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:49.994350 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:49.994664 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:49.995019 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 20:52:49.995302 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":551,"threads_started":1}
I20260926 20:52:49.997051 15533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:49.997287 16047 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 FOLLOWER]: Refusing update from remote peer 9d6c916c8ec441e6be3173bb0f696d38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 20:52:49.998152 15907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: NON_VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:50.002087 15908 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0a04345301c847df90874e0f9cc23c84 (127.8.249.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: NON_VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: true } } }
I20260926 20:52:50.002935 15964 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0a04345301c847df90874e0f9cc23c84 (127.8.249.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: NON_VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: true } } }
I20260926 20:52:50.005722 15908 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.249.188:35777
I20260926 20:52:50.006316 15911 master.cc:537] Master@127.8.249.188:35777 shutting down...
I20260926 20:52:50.006448 15908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: NON_VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: true } } }
I20260926 20:52:50.006771 15908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:50.006496 15910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: NON_VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: true } } }
I20260926 20:52:50.007504 15910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:50.009402 15992 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: NON_VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: true } } }
I20260926 20:52:50.009677 15992 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:52:50.016592 16047 raft_consensus.cc:858] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } }
W20260926 20:52:50.016803 16047 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } }
I20260926 20:52:50.029762 16047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER aefc04b1a7444d8fbe73b7a2388d47d1 (127.8.249.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } } }
I20260926 20:52:50.044747 16106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } } }
I20260926 20:52:50.045130 16106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 20:52:50.045657 16078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d6c916c8ec441e6be3173bb0f696d38. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } } }
I20260926 20:52:50.045887 16078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 20:52:50.046408 16102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: NON_VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: true } } }
I20260926 20:52:50.046659 16102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 20:52:50.047354 15910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: NON_VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: true } } }
I20260926 20:52:50.047663 15910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: This master's current role is: LEADER
W20260926 20:52:50.048529 15334 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 20:52:50.048941 15911 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260926 20:52:50.049563 15911 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260926 20:52:50.049775 15911 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: stopping tablet replica
W20260926 20:52:50.053465 15334 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 -> Peer 0a04345301c847df90874e0f9cc23c84 (127.8.249.188:35777): Couldn't send request to peer 0a04345301c847df90874e0f9cc23c84. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:52:50.067803 15911 master.cc:552] Master@127.8.249.188:35777 shutdown complete.
I20260926 20:52:50.067921 15911 master_runner.cc:305] Clearing existing system tablet
I20260926 20:52:50.068267 15911 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:50.068394 15911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:50.070632 15911 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 20:52:50.072402 16109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:50.072788 15911 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:50.073040 15911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 20:52:50.073134 15911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/wal
uuid: "0a04345301c847df90874e0f9cc23c84"
format_stamp: "Formatted at 2026-09-26 20:52:49 on dist-test-slave-2dg3"
server_key: "89acb5b7cc970a0ac422b0078be972ea"
server_key_iv: "4ad6ec9d67cc9df1f0d84629048fff86"
server_key_version: "encryptionkey@0"
I20260926 20:52:50.076779 15911 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:50.081895 15911 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:52:50.082082 15911 log.cc:1196] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 20:52:50.082525 15911 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: Deleting consensus metadata
I20260926 20:52:50.082777 15911 master_runner.cc:315] Copying system tablet from 127.8.249.190:44135
I20260926 20:52:50.091459 15911 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: tablet copy: Beginning tablet copy session from remote peer at address 127.8.249.190:44135
I20260926 20:52:50.105319 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:50.092600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.8.249.190:44135)
0926 20:52:50.092825 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:50.092831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:50.092905 (+ 74us) server_negotiation.cc:407] Connection header received
0926 20:52:50.093232 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:50.093238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:50.093343 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:50.093473 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:50.094502 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.095539 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.097069 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.097487 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.097633 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:50.099611 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:50.099617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:50.099622 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:50.099662 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:50.102569 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:50.103342 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:50.103349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:50.103353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:50.103534 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:50.103912 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:50.103919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:50.103922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:50.104194 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:50.104342 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:50.104728 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 20:52:50.104951 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 20:52:50.105418 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:50.092285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:38532)
0926 20:52:50.092797 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:50.092819 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:50.093075 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:50.093489 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:50.093493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:50.094053 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:50.094323 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:50.094336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.095744 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.095749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:50.096903 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:50.096915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.097638 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.097644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:50.097855 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:50.098550 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:50.098572 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:50.099414 (+ 842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:50.102705 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:50.102710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:50.102715 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:50.103226 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:50.103652 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:50.103658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:50.103662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:50.103807 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:50.104318 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:50.104327 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:50.104497 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 20:52:50.104637 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 20:52:50.104968 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":551,"threads_started":1}
I20260926 20:52:50.106343 15369 tablet_copy_service.cc:140] P 9d6c916c8ec441e6be3173bb0f696d38: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0a04345301c847df90874e0f9cc23c84 ({username='oryx', principal='oryx/127.8.249.188@KRBTEST.COM'} at 127.0.0.1:38532)
I20260926 20:52:50.106527 15369 tablet_copy_service.cc:161] P 9d6c916c8ec441e6be3173bb0f696d38: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0a04345301c847df90874e0f9cc23c84 at {username='oryx', principal='oryx/127.8.249.188@KRBTEST.COM'} at 127.0.0.1:38532: session id = 0a04345301c847df90874e0f9cc23c84-00000000000000000000000000000000
I20260926 20:52:50.110204 15369 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:52:50.112112 15911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:52:50.119016 15911 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: tablet copy: Starting download of 0 data blocks...
I20260926 20:52:50.119313 15911 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: tablet copy: Starting download of 1 WAL segments...
I20260926 20:52:50.121871 15911 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:52:50.126454 15369 tablet_copy_service.cc:342] P 9d6c916c8ec441e6be3173bb0f696d38: Request end of tablet copy session 0a04345301c847df90874e0f9cc23c84-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.249.188@KRBTEST.COM'} at 127.0.0.1:38532
I20260926 20:52:50.126617 15369 tablet_copy_service.cc:433] P 9d6c916c8ec441e6be3173bb0f696d38: ending tablet copy session 0a04345301c847df90874e0f9cc23c84-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0a04345301c847df90874e0f9cc23c84
I20260926 20:52:50.138350 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 15410
I20260926 20:52:50.138715 15911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:50.141276 16116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:50.148494 16117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:50.156055 15911 server_base.cc:1034] running on GCE node
W20260926 20:52:50.156410 16119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:50.156939 15911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:50.157678 15911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:50.158892 15911 hybrid_clock.cc:648] HybridClock initialized: now 1790455970158848 us; error 34 us; skew 500 ppm
I20260926 20:52:50.159186 15911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:50 dist-test-slave-2dg3 krb5kdc[15304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455970, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:50.161574 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:33315
--local_ip_for_outbound_sockets=127.8.249.129
--tserver_master_addrs=127.8.249.190:44135
--webserver_port=40951
--webserver_interface=127.8.249.129
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab
--principal=oryx/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.249.190:44135,127.8.249.189:46689
--tserver_master_addrs=127.8.249.190:44135,127.8.249.189:46689,127.8.249.188:35777 with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:52:50.166887 15911 init.cc:377] Logged in from keytab as oryx/127.8.249.188@KRBTEST.COM (short username oryx)
I20260926 20:52:50.168386 15911 webserver.cc:460] Webserver started at http://127.8.249.188:43679/ using document root <none> and password file <none>
I20260926 20:52:50.168752 15911 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:50.168848 15911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:50.172314 15911 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 20:52:50.174403 16127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:50.174845 15911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 20:52:50.174978 15911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/wal
uuid: "0a04345301c847df90874e0f9cc23c84"
format_stamp: "Formatted at 2026-09-26 20:52:49 on dist-test-slave-2dg3"
server_key: "89acb5b7cc970a0ac422b0078be972ea"
server_key_iv: "4ad6ec9d67cc9df1f0d84629048fff86"
server_key_version: "encryptionkey@0"
I20260926 20:52:50.175168 15911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:50.193449 15911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:50.197558 15911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:50.217378 15911 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.188:35777
I20260926 20:52:50.217446 16178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.188:35777 every 8 connection(s)
I20260926 20:52:50.218546 15911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/master-2/data/info.pb
I20260926 20:52:50.223791 16179 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:52:50.227816 16179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: Bootstrap starting.
I20260926 20:52:50.243760 16179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:52:50.244349 16179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: Bootstrap complete.
I20260926 20:52:50.245455 16179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: NON_VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: true } }
I20260926 20:52:50.245792 16179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0a04345301c847df90874e0f9cc23c84, State: Initialized, Role: LEARNER
I20260926 20:52:50.246054 16179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: NON_VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: true } }
I20260926 20:52:50.246954 16182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: NON_VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: true } } }
I20260926 20:52:50.247385 16182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:52:50.247382 16179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:50.259888 16193 catalog_manager.cc:1259] Loaded cluster ID: 986a280ad246442aafa35051eae5c0c7
I20260926 20:52:50.260006 16193 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: loading cluster ID for follower catalog manager: success
I20260926 20:52:50.263484 16193 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: acquiring CA information for follower catalog manager: success
I20260926 20:52:50.266130 16193 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:52:50.503726 15908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: NON_VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: true } } }
I20260926 20:52:50.504051 15908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:50.510975 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:50.504124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:35777 (local address 127.0.0.1:46290)
0926 20:52:50.504733 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:50.504755 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:50.504900 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:50.505348 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:50.505351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:50.505372 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:50.505542 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:50.505550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.507298 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.507302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:50.508610 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:50.508623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.510246 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.510253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:50.510439 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:50.510471 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 20:52:50.510586 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:52:50.510681 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":330,"threads_started":1}
I20260926 20:52:50.511096 16195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:50.504247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46290 (local address 127.8.249.188:35777)
0926 20:52:50.504987 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:50.504993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:50.505013 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:52:50.505087 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:50.505092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:50.505191 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:50.505333 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:50.505682 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.507154 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.508762 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.510015 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.510142 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:50.510398 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:50.510628 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 20:52:50.510765 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":463,"threads_started":1}
I20260926 20:52:50.511831 16154 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260926 20:52:50.512274 16182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d6c916c8ec441e6be3173bb0f696d38. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: NON_VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: true } } }
I20260926 20:52:50.512538 16182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:52:50.512835 15908 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38: attempting to promote NON_VOTER 0a04345301c847df90874e0f9cc23c84 to VOTER
I20260926 20:52:50.513520 15908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } }
I20260926 20:52:50.515403 16154 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 LEARNER]: Refusing update from remote peer 9d6c916c8ec441e6be3173bb0f696d38: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 20:52:50.515503 15964 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 FOLLOWER]: Refusing update from remote peer 9d6c916c8ec441e6be3173bb0f696d38: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 20:52:50.519081 15910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 20:52:50.519320 15907 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 20:52:50.521703 15910 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0a04345301c847df90874e0f9cc23c84 (127.8.249.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } } }
I20260926 20:52:50.522281 15964 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0a04345301c847df90874e0f9cc23c84 (127.8.249.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } } }
I20260926 20:52:50.525532 15910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } } }
I20260926 20:52:50.525854 15910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:50.525695 15908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } } }
I20260926 20:52:50.526082 15908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:50.536706 15908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } } }
I20260926 20:52:50.537020 15908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:50.538033 16154 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0a04345301c847df90874e0f9cc23c84 (127.8.249.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } } }
I20260926 20:52:50.538709 16198 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } } }
I20260926 20:52:50.539011 16198 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:52:50.541149 16182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d6c916c8ec441e6be3173bb0f696d38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } } }
I20260926 20:52:50.541411 16182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:52:50.546087 16122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:50.546350 16122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:50.546448 16122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:50.555039 16122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:50.555217 16122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:50.555418 16122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:52:50.565045 16122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab
--principal=oryx/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:33315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=40951
--tserver_master_addrs=127.8.249.190:44135,127.8.249.189:46689,127.8.249.188:35777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:50.566401 16122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:50.567821 16122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:50.580701 16202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:50.580737 16203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:50.580881 16205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:50.581830 16122 server_base.cc:1034] running on GCE node
I20260926 20:52:50.582374 16122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:50.583280 16122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:50.584468 16122 hybrid_clock.cc:648] HybridClock initialized: now 1790455970584423 us; error 38 us; skew 500 ppm
I20260926 20:52:50.584776 16122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:50 dist-test-slave-2dg3 krb5kdc[15304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455970, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:50.588058 16122 init.cc:377] Logged in from keytab as oryx/127.8.249.129@KRBTEST.COM (short username oryx)
I20260926 20:52:50.589881 16122 webserver.cc:460] Webserver started at http://127.8.249.129:40951/ using document root <none> and password file <none>
I20260926 20:52:50.590595 16122 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:50.590754 16122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:50.596694 16122 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 20:52:50.599431 16212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:50.600751 16122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:50.600878 16122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "db22faab4b5d414fadbce62b9eec0d9e"
format_stamp: "Formatted at 2026-09-26 20:52:47 on dist-test-slave-2dg3"
server_key: "7618b31b8ef3ac326ce5d1d302047625"
server_key_iv: "209b4b94a13989f418129f23c97b1c25"
server_key_version: "encryptionkey@0"
I20260926 20:52:50.601827 16122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:50.624720 16122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:50.629130 16122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:50.629361 16122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:50.630281 16122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:50.631706 16122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:50.631798 16122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:50.631870 16122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:50.631924 16122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:50.664839 16122 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:33315
I20260926 20:52:50.664942 16324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:33315 every 8 connection(s)
I20260926 20:52:50.666240 16122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:52:50.667630 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 16122
I20260926 20:52:50.668061 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 15544
Sep 26 20:52:50 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455970, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.188@KRBTEST.COM
Sep 26 20:52:50 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.249.130@KRBTEST.COM
I20260926 20:52:50.700498 16195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:50.672473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:34699 (local address 127.8.249.188:35777)
0926 20:52:50.672735 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:50.672746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:50.672778 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:52:50.676642 (+ 3864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:50.676652 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:50.676764 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:50.676854 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:50.679746 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.681215 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.684538 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.685439 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.685560 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:50.691334 (+ 5774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:50.691344 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:50.691351 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:50.691406 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:50.695591 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:50.696861 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:50.696870 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:50.696876 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:50.697078 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:50.697753 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:50.697761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:50.697767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:50.698201 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:50.698364 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:50.699811 (+ 1447us) server_negotiation.cc:299] Negotiation successful
0926 20:52:50.700139 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 20:52:50.703437 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:50.671827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:35777 (local address 127.8.249.129:34699)
0926 20:52:50.672632 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:50.672697 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:50.676206 (+ 3509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:50.677502 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:50.677514 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:50.678504 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:50.679484 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:50.679516 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.681470 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.681479 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:50.684338 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:50.684358 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.686010 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.686019 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:50.686416 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:50.687336 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:50.687376 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:50.691028 (+ 3652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:50.695838 (+ 4810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:50.695851 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:50.695875 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:50.696688 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:50.697261 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:50.697269 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:50.697275 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:50.697598 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:50.698485 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:50.698501 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:50.699138 (+ 637us) client_negotiation.cc:769] Sending connection context
0926 20:52:50.701112 (+ 1974us) client_negotiation.cc:240] Negotiation successful
0926 20:52:50.701930 (+ 818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":971,"threads_started":1}
I20260926 20:52:50.705262 16325 heartbeater.cc:347] Connected to a master server at 127.8.249.188:35777
I20260926 20:52:50.705621 16325 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:50.706563 16325 heartbeater.cc:511] Master 127.8.249.188:35777 requested a full tablet report, sending...
I20260926 20:52:50.709323 16144 ts_manager.cc:194] Registered new tserver with Master: db22faab4b5d414fadbce62b9eec0d9e (127.8.249.129:33315)
Sep 26 20:52:50 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455970, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.189@KRBTEST.COM
I20260926 20:52:50.709986 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:50.671023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51788 (local address 127.8.249.130:42309)
0926 20:52:50.684081 (+ 13058us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:50.684094 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:50.684133 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:52:50.684475 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:50.684487 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:50.684781 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:50.687695 (+ 2914us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:50.687998 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.689827 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.692338 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.693484 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.700465 (+ 6981us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:50.700800 (+ 335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:50.700817 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:50.700840 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:50.700883 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:50.705216 (+ 4333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:50.707022 (+ 1806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:50.707032 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:50.707035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:50.707208 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:50.708041 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:50.708049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:50.708055 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:50.708516 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:50.708841 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:50.709209 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 20:52:50.709503 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12825,"thread_start_us":13179,"threads_started":1}
I20260926 20:52:50.711452 15667 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 26 20:52:50 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455970, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:52:50.717475 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:50.673887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:53829 (local address 127.8.249.189:46689)
0926 20:52:50.696791 (+ 22904us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:50.696801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:50.697716 (+ 915us) server_negotiation.cc:407] Connection header received
0926 20:52:50.697797 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:50.697804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:50.697910 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:50.698023 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:50.699579 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.700733 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.705676 (+ 4943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.706539 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.706701 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:50.710422 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:50.710428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:50.710433 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:50.710478 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:50.714252 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:50.715192 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:50.715201 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:50.715208 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:50.715425 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:50.716030 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:50.716038 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:50.716042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:50.716378 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:50.716518 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:50.716888 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 20:52:50.717100 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22690,"thread_start_us":22904,"threads_started":1}
I20260926 20:52:50.717496 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:50.673367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46689 (local address 127.8.249.129:53829)
0926 20:52:50.697388 (+ 24021us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:50.697420 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:50.697580 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:50.698264 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:50.698272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:50.699006 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:50.699368 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:50.699383 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.701237 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.701243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:50.705512 (+ 4269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:50.705525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.706758 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.706765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:50.707068 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:50.707724 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:50.707753 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:50.710230 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:50.714441 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:50.714447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:50.714451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:50.715021 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:50.715626 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:50.715634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:50.715641 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:50.715851 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:50.716484 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:50.716489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:50.716641 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 20:52:50.716779 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 20:52:50.717072 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23865,"mutex_wait_us":25,"thread_start_us":27492,"threads_started":1}
I20260926 20:52:50.719383 16327 heartbeater.cc:347] Connected to a master server at 127.8.249.189:46689
I20260926 20:52:50.719509 16327 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:50.719986 16327 heartbeater.cc:511] Master 127.8.249.189:46689 requested a full tablet report, sending...
I20260926 20:52:50.721968 15954 ts_manager.cc:194] Registered new tserver with Master: db22faab4b5d414fadbce62b9eec0d9e (127.8.249.129:33315)
I20260926 20:52:50.724228 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:50.671068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:54381 (local address 127.8.249.190:44135)
0926 20:52:50.702377 (+ 31309us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:50.702385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:50.702407 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:52:50.702476 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:50.702481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:50.702570 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:50.702698 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:50.704467 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.706110 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.710585 (+ 4475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.711348 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.711524 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:50.715613 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:50.715618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:50.715622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:50.715653 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:50.720850 (+ 5197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:50.721849 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:50.721855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:50.721859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:50.722066 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:50.722553 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:50.722558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:50.722562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:50.722993 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:50.723163 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:50.723580 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 20:52:50.723795 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31143,"thread_start_us":31023,"threads_started":1}
I20260926 20:52:50.724228 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:50.670164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.129:54381)
0926 20:52:50.677394 (+ 7230us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:50.677419 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:50.677516 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:50.702748 (+ 25232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:50.702753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:50.703453 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:50.703751 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:50.703762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.706266 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.706271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:50.707982 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:50.707998 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:50.711550 (+ 3552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:50.711556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:50.711753 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:50.712320 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:50.712342 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:50.715385 (+ 3043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:50.721052 (+ 5667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:50.721060 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:50.721067 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:50.721661 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:50.722257 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:50.722262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:50.722266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:50.722423 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:50.723129 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:50.723135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:50.723302 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 20:52:50.723461 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 20:52:50.723760 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":4,"thread_start_us":928,"threads_started":1}
I20260926 20:52:50.725551 16326 heartbeater.cc:347] Connected to a master server at 127.8.249.190:44135
I20260926 20:52:50.725711 16326 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:50.726150 16326 heartbeater.cc:511] Master 127.8.249.190:44135 requested a full tablet report, sending...
I20260926 20:52:50.727347 15349 ts_manager.cc:194] Re-registered known tserver with Master: db22faab4b5d414fadbce62b9eec0d9e (127.8.249.129:33315)
I20260926 20:52:50.728281 15349 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:54381
I20260926 20:52:50.802371 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 15544
I20260926 20:52:50.819090 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:42309
--local_ip_for_outbound_sockets=127.8.249.130
--tserver_master_addrs=127.8.249.190:44135
--webserver_port=45447
--webserver_interface=127.8.249.130
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab
--principal=oryx/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.249.190:44135,127.8.249.189:46689
--tserver_master_addrs=127.8.249.190:44135,127.8.249.189:46689,127.8.249.188:35777 with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:51.147738 16337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:51.148102 16337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:51.148263 16337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:51.155932 16337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:51.156085 16337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:51.156261 16337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:52:51.166043 16337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab
--principal=oryx/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:42309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=45447
--tserver_master_addrs=127.8.249.190:44135,127.8.249.189:46689,127.8.249.188:35777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:51.167518 16337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:51.168908 16337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:51.182381 16342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:51.182459 16345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:51.182448 16343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:51.183456 16337 server_base.cc:1034] running on GCE node
I20260926 20:52:51.184155 16337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:51.185240 16337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:51.186473 16337 hybrid_clock.cc:648] HybridClock initialized: now 1790455971186440 us; error 38 us; skew 500 ppm
I20260926 20:52:51.186882 16337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:51 dist-test-slave-2dg3 krb5kdc[15304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455971, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:51.190081 16337 init.cc:377] Logged in from keytab as oryx/127.8.249.130@KRBTEST.COM (short username oryx)
I20260926 20:52:51.191920 16337 webserver.cc:460] Webserver started at http://127.8.249.130:45447/ using document root <none> and password file <none>
I20260926 20:52:51.192670 16337 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:51.192837 16337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:51.198432 16337 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 20:52:51.201344 16352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:51.202876 16337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:51.203060 16337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "00012a67b7e844bd8563f8cfaa0013b1"
format_stamp: "Formatted at 2026-09-26 20:52:47 on dist-test-slave-2dg3"
server_key: "4c3d0eae8311327d4f910187c84316e3"
server_key_iv: "bc1d0d49ea647769aa0ee74c6728d16c"
server_key_version: "encryptionkey@0"
I20260926 20:52:51.204131 16337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:51.231288 16337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:51.235163 16337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:51.235432 16337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:51.236371 16337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:51.237694 16337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:51.237816 16337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:51.237937 16337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:51.238021 16337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:51.271767 16337 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:42309
I20260926 20:52:51.271853 16464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:42309 every 8 connection(s)
I20260926 20:52:51.273242 16337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:52:51.280129 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 16337
I20260926 20:52:51.280440 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 15677
Sep 26 20:52:51 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455971, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
Sep 26 20:52:51 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455971, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.189@KRBTEST.COM
Sep 26 20:52:51 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.249.131@KRBTEST.COM
Sep 26 20:52:51 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455971, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.188@KRBTEST.COM
I20260926 20:52:51.302366 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.277465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:42323 (local address 127.8.249.190:44135)
0926 20:52:51.279842 (+ 2377us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:51.279851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:51.279877 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:52:51.281366 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:51.281371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:51.281473 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:51.281557 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:51.284932 (+ 3375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.286000 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.288137 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.288785 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.288887 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.294535 (+ 5648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:51.294544 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:51.294550 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:51.294609 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:51.298266 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.299325 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.299330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.299335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.299497 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.300054 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.300062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.300065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.300451 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:51.300610 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:51.301672 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0926 20:52:51.301971 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2208,"thread_start_us":2129,"threads_started":1}
I20260926 20:52:51.302647 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.276247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:59213 (local address 127.8.249.189:46689)
0926 20:52:51.276748 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:51.276755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:51.279442 (+ 2687us) server_negotiation.cc:407] Connection header received
0926 20:52:51.281447 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:51.281452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:51.281530 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:51.281626 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:51.284212 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.285341 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.288144 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.288784 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.288899 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.294904 (+ 6005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:51.294911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:51.294916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:51.294957 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:51.298985 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.299838 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.299843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.299846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.299985 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.300703 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.300708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.300711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.301084 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:51.301236 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:51.302020 (+ 784us) server_negotiation.cc:299] Negotiation successful
0926 20:52:51.302242 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":228,"threads_started":1}
I20260926 20:52:51.305063 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.276951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.130:42323)
0926 20:52:51.277986 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:51.278051 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:51.281023 (+ 2972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:51.282015 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:51.282028 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:51.284510 (+ 2482us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:51.284738 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.284750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.286199 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.286204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.287970 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.287985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.289211 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.289218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.289622 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.290653 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:51.290696 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:51.294216 (+ 3520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:51.298435 (+ 4219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:51.298444 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:51.298466 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:51.299173 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:51.299629 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:51.299635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:51.299641 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:51.299904 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:51.300704 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:51.300713 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:51.301184 (+ 471us) client_negotiation.cc:769] Sending connection context
0926 20:52:51.303203 (+ 2019us) client_negotiation.cc:240] Negotiation successful
0926 20:52:51.303633 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":286,"spinlock_wait_cycles":75776,"thread_start_us":425,"threads_started":1}
I20260926 20:52:51.305085 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.276971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46689 (local address 127.8.249.130:59213)
0926 20:52:51.279343 (+ 2372us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:51.279363 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:51.281023 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:51.282015 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:51.282028 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:51.283052 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:51.283970 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.283997 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.285560 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.285565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.287970 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.287985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.289211 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.289218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.289624 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.290763 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:51.290833 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:51.294663 (+ 3830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:51.299137 (+ 4474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:51.299142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:51.299146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:51.299714 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:51.300413 (+ 699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:51.300418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:51.300421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:51.300586 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:51.301300 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:51.301307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:51.301492 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 20:52:51.301691 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 20:52:51.303472 (+ 1781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2243,"mutex_wait_us":205,"spinlock_wait_cycles":1792,"thread_start_us":1893,"threads_started":1}
I20260926 20:52:51.306989 16467 heartbeater.cc:347] Connected to a master server at 127.8.249.189:46689
I20260926 20:52:51.307322 16467 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:51.307489 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.280436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:35777 (local address 127.8.249.130:58931)
0926 20:52:51.283403 (+ 2967us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:51.283426 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:51.283561 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:51.290920 (+ 7359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:51.290927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:51.291570 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:51.291962 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.292030 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.293830 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.293838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.295492 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.295505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.296583 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.296591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.296792 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.297380 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:51.297411 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:51.300461 (+ 3050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:51.304331 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:51.304336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:51.304339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:51.304871 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:51.305323 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:51.305331 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:51.305335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:51.305502 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:51.306439 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:51.306446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:51.306594 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 20:52:51.306738 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 20:52:51.307083 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2814,"thread_start_us":450,"threads_started":1}
I20260926 20:52:51.308136 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.281349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.8.249.131:39939)
0926 20:52:51.282757 (+ 1408us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:51.282767 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:51.283448 (+ 681us) server_negotiation.cc:407] Connection header received
0926 20:52:51.283519 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:51.283528 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:51.283783 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:51.283946 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:51.286861 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.288943 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.291182 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.292046 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.292192 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.298600 (+ 6408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:51.298618 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:51.298640 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:51.298690 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:51.303646 (+ 4956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.304546 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.304554 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.304558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.304721 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.305287 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.305291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.305294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.306224 (+ 930us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:51.307163 (+ 939us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:51.307476 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 20:52:51.307795 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1234,"thread_start_us":1057,"threads_started":1}
I20260926 20:52:51.308305 16467 heartbeater.cc:511] Master 127.8.249.189:46689 requested a full tablet report, sending...
I20260926 20:52:51.308595 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.279638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:58931 (local address 127.8.249.188:35777)
0926 20:52:51.290468 (+ 10830us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:51.290475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:51.290497 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:52:51.290569 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:51.290576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:51.290674 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:51.292183 (+ 1509us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:51.292383 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.293674 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.295693 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.296362 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.297064 (+ 702us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.300687 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:51.300693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:51.300697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:51.300734 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:51.304184 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.305049 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.305056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.305062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.305202 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.305717 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.305722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.305725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.306103 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:51.307838 (+ 1735us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:51.307969 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 20:52:51.308205 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10675,"thread_start_us":10562,"threads_started":1}
I20260926 20:52:51.309566 15800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:51.309561 16465 heartbeater.cc:347] Connected to a master server at 127.8.249.188:35777
I20260926 20:52:51.309707 16465 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:51.309811 16466 heartbeater.cc:347] Connected to a master server at 127.8.249.190:44135
I20260926 20:52:51.309981 16466 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:51.310325 16465 heartbeater.cc:511] Master 127.8.249.188:35777 requested a full tablet report, sending...
I20260926 20:52:51.310515 16466 heartbeater.cc:511] Master 127.8.249.190:44135 requested a full tablet report, sending...
I20260926 20:52:51.311406 15954 ts_manager.cc:194] Registered new tserver with Master: 00012a67b7e844bd8563f8cfaa0013b1 (127.8.249.130:42309)
I20260926 20:52:51.311550 16144 ts_manager.cc:194] Registered new tserver with Master: 00012a67b7e844bd8563f8cfaa0013b1 (127.8.249.130:42309)
I20260926 20:52:51.311792 15349 ts_manager.cc:194] Re-registered known tserver with Master: 00012a67b7e844bd8563f8cfaa0013b1 (127.8.249.130:42309)
I20260926 20:52:51.313162 15349 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:42323
I20260926 20:52:51.392241 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 15677
I20260926 20:52:51.409183 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:39939
--local_ip_for_outbound_sockets=127.8.249.131
--tserver_master_addrs=127.8.249.190:44135
--webserver_port=40765
--webserver_interface=127.8.249.131
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab
--principal=oryx/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.249.190:44135,127.8.249.189:46689
--tserver_master_addrs=127.8.249.190:44135,127.8.249.189:46689,127.8.249.188:35777 with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:52:51.730715 16326 heartbeater.cc:503] Master 127.8.249.190:44135 was elected leader, sending a full tablet report...
W20260926 20:52:51.741127 16478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:51.741477 16478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:51.741667 16478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:51.749310 16478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:51.749475 16478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:51.749655 16478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:52:51.760141 16478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab
--principal=oryx/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:39939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=40765
--tserver_master_addrs=127.8.249.190:44135,127.8.249.189:46689,127.8.249.188:35777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:51.761552 16478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:51.763047 16478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:51.776724 16484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:51.776772 16486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:51.776684 16483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:51.780228 16478 server_base.cc:1034] running on GCE node
I20260926 20:52:51.780825 16478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:51.781744 16478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:51.782956 16478 hybrid_clock.cc:648] HybridClock initialized: now 1790455971782908 us; error 43 us; skew 500 ppm
I20260926 20:52:51.783267 16478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:51 dist-test-slave-2dg3 krb5kdc[15304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455971, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:51.786422 16478 init.cc:377] Logged in from keytab as oryx/127.8.249.131@KRBTEST.COM (short username oryx)
I20260926 20:52:51.787878 16478 webserver.cc:460] Webserver started at http://127.8.249.131:40765/ using document root <none> and password file <none>
I20260926 20:52:51.788614 16478 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:51.788764 16478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:51.794041 16478 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 20:52:51.796646 16493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:51.797890 16478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:52:51.798074 16478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "f21ee59a5d9f4b7f867c0f978165a969"
format_stamp: "Formatted at 2026-09-26 20:52:48 on dist-test-slave-2dg3"
server_key: "d008e7529d38e179b17f8b280db7fbb8"
server_key_iv: "9a4c4ec8cdc6c072f7b464095bdd3616"
server_key_version: "encryptionkey@0"
I20260926 20:52:51.799105 16478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:51.815435 16478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:51.819442 16478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:51.819713 16478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:51.820662 16478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:52:51.822031 16478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:51.822165 16478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:51.822290 16478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:51.822376 16478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:51.855944 16478 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:39939
I20260926 20:52:51.856027 16605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:39939 every 8 connection(s)
I20260926 20:52:51.857357 16478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:52:51.860008 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 16478
Sep 26 20:52:51 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455971, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.188@KRBTEST.COM
Sep 26 20:52:51 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455971, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:52:51.904099 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.860172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:54837 (local address 127.8.249.188:35777)
0926 20:52:51.875244 (+ 15072us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:51.875253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:51.880408 (+ 5155us) server_negotiation.cc:407] Connection header received
0926 20:52:51.882542 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:51.882547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:51.882656 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:51.882777 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:51.885768 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.887260 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.890033 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.890695 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.890810 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.896229 (+ 5419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:51.896238 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:51.896246 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:51.896292 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:51.900143 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.901282 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.901287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.901291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.901448 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.901989 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.901994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.901998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.902342 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:51.902469 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:51.903522 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0926 20:52:51.903739 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14897,"thread_start_us":14768,"threads_started":1}
I20260926 20:52:51.905556 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.860825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.131:57451)
0926 20:52:51.883077 (+ 22252us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:51.883104 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:51.883219 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:51.883770 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:51.883774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:51.884368 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:51.888589 (+ 4221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.888602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.890009 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.890017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.891558 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.891570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.892701 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.892707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.892899 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.893587 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:51.893609 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:51.896208 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:51.900318 (+ 4110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:51.900327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:51.900345 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:51.901128 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:51.901582 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:51.901586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:51.901589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:51.901835 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:51.902559 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:51.902567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:51.903097 (+ 530us) client_negotiation.cc:769] Sending connection context
0926 20:52:51.903412 (+ 315us) client_negotiation.cc:240] Negotiation successful
0926 20:52:51.903915 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22080,"mutex_wait_us":3948,"spinlock_wait_cycles":56064,"thread_start_us":18792,"threads_started":1}
I20260926 20:52:51.905579 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.860825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:35777 (local address 127.8.249.131:54837)
0926 20:52:51.880215 (+ 19390us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:51.880267 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:51.882240 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:51.883292 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:51.883307 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:51.884336 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:51.885502 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.885536 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.887507 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.887515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.889873 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.889889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.891156 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.891163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.891446 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.892328 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:51.892360 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:51.895899 (+ 3539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:51.900318 (+ 4419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:51.900327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:51.900345 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:51.901128 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:51.901596 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:51.901600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:51.901603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:51.901871 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:51.902559 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:51.902567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:51.903095 (+ 528us) client_negotiation.cc:769] Sending connection context
0926 20:52:51.904507 (+ 1412us) client_negotiation.cc:240] Negotiation successful
0926 20:52:51.904794 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19059,"mutex_wait_us":42,"spinlock_wait_cycles":30336,"thread_start_us":21808,"threads_started":1}
I20260926 20:52:51.906718 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.861677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:57451 (local address 127.8.249.190:44135)
0926 20:52:51.879266 (+ 17589us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:51.879275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:51.883331 (+ 4056us) server_negotiation.cc:407] Connection header received
0926 20:52:51.883394 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:51.883398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:51.883475 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:51.883581 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:51.888771 (+ 5190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.889818 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.891727 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.892329 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.892430 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.896412 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:51.896421 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:51.896428 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:51.896468 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:51.900141 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.901282 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.901287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.901291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.901458 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.901989 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.901994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.901998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.902358 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:51.902485 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:51.906078 (+ 3593us) server_negotiation.cc:299] Negotiation successful
0926 20:52:51.906338 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17426,"thread_start_us":17318,"threads_started":1}
I20260926 20:52:51.907857 16606 heartbeater.cc:347] Connected to a master server at 127.8.249.188:35777
I20260926 20:52:51.908278 16606 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:51.909372 16606 heartbeater.cc:511] Master 127.8.249.188:35777 requested a full tablet report, sending...
I20260926 20:52:51.907888 16607 heartbeater.cc:347] Connected to a master server at 127.8.249.190:44135
I20260926 20:52:51.910429 16607 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:51.910943 16607 heartbeater.cc:511] Master 127.8.249.190:44135 requested a full tablet report, sending...
I20260926 20:52:51.911362 16144 ts_manager.cc:194] Registered new tserver with Master: f21ee59a5d9f4b7f867c0f978165a969 (127.8.249.131:39939)
I20260926 20:52:51.911916 15349 ts_manager.cc:194] Re-registered known tserver with Master: f21ee59a5d9f4b7f867c0f978165a969 (127.8.249.131:39939)
I20260926 20:52:51.913015 15349 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:57451
Sep 26 20:52:51 dist-test-slave-2dg3 krb5kdc[15304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455971, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.189@KRBTEST.COM
I20260926 20:52:51.926849 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.884564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46689 (local address 127.8.249.131:56807)
0926 20:52:51.888196 (+ 3632us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:51.888224 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:51.888381 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:51.888901 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:51.888909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:51.889698 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:51.890184 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.890203 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.892722 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.892731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.901682 (+ 8951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.901703 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.915036 (+ 13333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.915047 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.915329 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.916032 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:51.916060 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:51.919289 (+ 3229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:51.923466 (+ 4177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:51.923472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:51.923477 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:51.924293 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:51.924727 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:51.924734 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:51.924741 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:51.924917 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:51.925750 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:51.925755 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:51.925922 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 20:52:51.926064 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 20:52:51.926394 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3499,"thread_start_us":3264,"threads_started":1}
I20260926 20:52:51.927670 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.883545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:56807 (local address 127.8.249.189:46689)
0926 20:52:51.887793 (+ 4248us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:51.887799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:51.888529 (+ 730us) server_negotiation.cc:407] Connection header received
0926 20:52:51.888608 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:51.888616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:51.888715 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:51.890381 (+ 1666us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:51.890581 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.892447 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.902186 (+ 9739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.902932 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.903046 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.919581 (+ 16535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:51.919598 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:51.919605 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:51.919674 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:51.923314 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.924435 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.924440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.924444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.924592 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.925056 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.925063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.925067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.925499 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:51.925688 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:51.927041 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0926 20:52:51.927279 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":4117,"thread_start_us":3982,"threads_started":1}
I20260926 20:52:51.928539 16608 heartbeater.cc:347] Connected to a master server at 127.8.249.189:46689
I20260926 20:52:51.928679 16608 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:51.929098 16608 heartbeater.cc:511] Master 127.8.249.189:46689 requested a full tablet report, sending...
I20260926 20:52:51.930249 15954 ts_manager.cc:194] Registered new tserver with Master: f21ee59a5d9f4b7f867c0f978165a969 (127.8.249.131:39939)
I20260926 20:52:51.930478 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.903471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.8.249.190:44135)
0926 20:52:51.908051 (+ 4580us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:51.908057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:51.911463 (+ 3406us) server_negotiation.cc:407] Connection header received
0926 20:52:51.913577 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:51.913582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:51.913674 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:51.913745 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:51.916081 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.916852 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.918273 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.918738 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.918866 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.923069 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:51.923078 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:51.923085 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:51.923123 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:51.926478 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.927485 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.927490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.927494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.927620 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.928218 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.928222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.928225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.928582 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:51.928921 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:51.929572 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 20:52:51.930053 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4475,"thread_start_us":14680,"threads_started":1}
I20260926 20:52:51.939538 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.896628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.8.249.189:46689)
0926 20:52:51.922692 (+ 26064us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:51.922698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:51.922715 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:52:51.922764 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:51.922771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:51.922878 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:51.925050 (+ 2172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:51.925281 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.926549 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.928522 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.929148 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.929892 (+ 744us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.932427 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:51.932434 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:51.932442 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:51.932480 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:51.935813 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.936683 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.936687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.936690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.936843 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.937285 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.937290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.937294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.937618 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:51.938762 (+ 1144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:51.938907 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 20:52:51.939146 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":25926,"thread_start_us":25332,"threads_started":1}
I20260926 20:52:51.941104 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.911579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.8.249.188:35777)
0926 20:52:51.912050 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:51.912055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:51.922631 (+ 10576us) server_negotiation.cc:407] Connection header received
0926 20:52:51.922907 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:51.922927 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:51.923012 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:51.923076 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:51.925159 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.926293 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.929853 (+ 3560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.930472 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.930565 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.934219 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:51.934224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:51.934227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:51.934257 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:51.937657 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.938426 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.938431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.938434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.938605 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:51.939044 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:51.939051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:51.939058 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:51.939375 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:51.940071 (+ 696us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:51.940577 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 20:52:51.940801 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350}
I20260926 20:52:51.935889 15349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:52:51.947515 15349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:52:51.986774 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.967865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.8.249.129:33315)
0926 20:52:51.979407 (+ 11542us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:51.979420 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:51.979943 (+ 523us) server_negotiation.cc:407] Connection header received
0926 20:52:51.980283 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:51.980292 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:51.980533 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:51.980723 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:51.981262 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.982206 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.984599 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.985712 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.985856 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.986076 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:51.986256 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 20:52:51.986442 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11268,"thread_start_us":11112,"threads_started":1}
I20260926 20:52:51.987587 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.967646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:33315 (local address 127.0.0.1:41642)
0926 20:52:51.979838 (+ 12192us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:51.979853 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:51.980070 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:51.980718 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:51.980722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:51.980745 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:51.981107 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.981116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.982348 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.982354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.984447 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.984461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.985873 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.985879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.986033 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.986069 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 20:52:51.987103 (+ 1034us) client_negotiation.cc:240] Negotiation successful
0926 20:52:51.987245 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12082,"thread_start_us":11942,"threads_started":1}
I20260926 20:52:51.987897 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.966175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.8.249.130:42309)
0926 20:52:51.977923 (+ 11748us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:51.977934 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:51.978754 (+ 820us) server_negotiation.cc:407] Connection header received
0926 20:52:51.978918 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:51.978928 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:51.979239 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:51.979486 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:51.981027 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.983004 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.985104 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.986677 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.986837 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.987203 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:51.987453 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 20:52:51.987622 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10857,"thread_start_us":11468,"threads_started":1}
I20260926 20:52:51.988238 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.965928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:42309 (local address 127.0.0.1:51802)
0926 20:52:51.978411 (+ 12483us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:51.978436 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:51.978606 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:51.979750 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:51.979758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:51.979792 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:51.980740 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.980758 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.983165 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.983169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.984956 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.984972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.987081 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.987090 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.987255 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.987292 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:52:51.987404 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 20:52:51.987528 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12324,"thread_start_us":12200,"threads_started":1}
I20260926 20:52:51.992019 16400 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4f9573726848598dbfbc4ec3028216 (DEFAULT_TABLE table=test-table [id=e123b77badc245e996389757189b6a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:51.992841 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.965492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:39939 (local address 127.0.0.1:60922)
0926 20:52:51.966051 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:51.966078 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:51.966219 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:51.984013 (+ 17794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:51.984018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:51.984044 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:51.984482 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.984493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.988229 (+ 3736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.988236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.989846 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:51.989859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.991980 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.991987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:51.992204 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.992238 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:52:51.992336 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:52:51.992472 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":838,"threads_started":1}
I20260926 20:52:51.993026 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:51.965826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60922 (local address 127.8.249.131:39939)
0926 20:52:51.983444 (+ 17618us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:51.983453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:51.983481 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:52:51.983566 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:51.983573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:51.983800 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:51.983985 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:51.986206 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.988102 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.990006 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:51.991740 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:51.991878 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:51.992071 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:51.992392 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 20:52:51.992559 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17450,"thread_start_us":17345,"threads_started":1}
I20260926 20:52:51.992621 16260 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4f9573726848598dbfbc4ec3028216 (DEFAULT_TABLE table=test-table [id=e123b77badc245e996389757189b6a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:51.993773 16260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4f9573726848598dbfbc4ec3028216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:51.993777 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4f9573726848598dbfbc4ec3028216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:51.999287 16541 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4f9573726848598dbfbc4ec3028216 (DEFAULT_TABLE table=test-table [id=e123b77badc245e996389757189b6a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:52.000746 16541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4f9573726848598dbfbc4ec3028216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:52.006057 16642 tablet_bootstrap.cc:492] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1: Bootstrap starting.
I20260926 20:52:52.009542 16642 tablet_bootstrap.cc:654] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:52.010701 16642 log.cc:824] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:52.014441 16643 tablet_bootstrap.cc:492] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e: Bootstrap starting.
I20260926 20:52:52.018175 16643 tablet_bootstrap.cc:654] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:52.018661 16642 tablet_bootstrap.cc:492] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1: No bootstrap required, opened a new log
I20260926 20:52:52.019019 16642 ts_tablet_manager.cc:1397] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260926 20:52:52.019717 16643 log.cc:824] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:52.023320 16643 tablet_bootstrap.cc:492] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e: No bootstrap required, opened a new log
I20260926 20:52:52.023593 16643 ts_tablet_manager.cc:1397] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 20:52:52.023717 16645 tablet_bootstrap.cc:492] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969: Bootstrap starting.
I20260926 20:52:52.027340 16645 tablet_bootstrap.cc:654] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:52.028576 16645 log.cc:824] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:52.028618 16642 raft_consensus.cc:348] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:52.029035 16642 raft_consensus.cc:374] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:52.029165 16642 raft_consensus.cc:729] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00012a67b7e844bd8563f8cfaa0013b1, State: Initialized, Role: FOLLOWER
I20260926 20:52:52.030020 16642 consensus_queue.cc:260] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:52.031920 16466 heartbeater.cc:503] Master 127.8.249.190:44135 was elected leader, sending a full tablet report...
I20260926 20:52:52.032563 16642 ts_tablet_manager.cc:1428] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 20:52:52.031556 16643 raft_consensus.cc:348] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:52.033488 16643 raft_consensus.cc:374] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:52.033653 16643 raft_consensus.cc:729] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db22faab4b5d414fadbce62b9eec0d9e, State: Initialized, Role: FOLLOWER
I20260926 20:52:52.034099 16645 tablet_bootstrap.cc:492] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969: No bootstrap required, opened a new log
I20260926 20:52:52.034363 16645 ts_tablet_manager.cc:1397] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 20:52:52.034364 16643 consensus_queue.cc:260] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:52.038554 16643 ts_tablet_manager.cc:1428] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 20:52:52.044031 16645 raft_consensus.cc:348] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:52.044391 16645 raft_consensus.cc:374] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:52.044502 16645 raft_consensus.cc:729] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21ee59a5d9f4b7f867c0f978165a969, State: Initialized, Role: FOLLOWER
I20260926 20:52:52.045116 16645 consensus_queue.cc:260] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:52.046094 16607 heartbeater.cc:503] Master 127.8.249.190:44135 was elected leader, sending a full tablet report...
I20260926 20:52:52.046770 16645 ts_tablet_manager.cc:1428] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
W20260926 20:52:52.121910 16614 tablet.cc:2369] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:52:52.171574 16329 tablet.cc:2369] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:52:52.274940 16649 raft_consensus.cc:482] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:52.275266 16649 raft_consensus.cc:504] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:52.277179 16649 leader_election.cc:288] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21ee59a5d9f4b7f867c0f978165a969 (127.8.249.131:39939), 00012a67b7e844bd8563f8cfaa0013b1 (127.8.249.130:42309)
W20260926 20:52:52.278097 16468 tablet.cc:2369] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:52:52.285840 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:52.278245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:60561 (local address 127.8.249.131:39939)
0926 20:52:52.278487 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:52.278495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:52.279854 (+ 1359us) server_negotiation.cc:407] Connection header received
0926 20:52:52.280063 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:52.280068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:52.280139 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:52.280239 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:52.280911 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.281733 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.283547 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.284725 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.284881 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:52.284955 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:52.285443 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 20:52:52.285600 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 20:52:52.286013 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:52.277698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:33031 (local address 127.8.249.130:42309)
0926 20:52:52.277929 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:52.277938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:52.279162 (+ 1224us) server_negotiation.cc:407] Connection header received
0926 20:52:52.279372 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:52.279379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:52.279465 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:52.279561 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:52.280278 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.281174 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.283742 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.285079 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.285270 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:52.285340 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:52.285643 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 20:52:52.285764 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:52:52.286267 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:52.277640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:39939 (local address 127.8.249.129:60561)
0926 20:52:52.279752 (+ 2112us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:52.279769 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:52.279929 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:52.280404 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:52.280409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:52.280431 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:52.280734 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:52.280743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.281874 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.281879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:52.283385 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:52.283396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.284954 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.284960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:52.285201 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:52.285240 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 20:52:52.285404 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 20:52:52.285538 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1982,"mutex_wait_us":57,"thread_start_us":1842,"threads_started":1}
I20260926 20:52:52.286715 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:52.277579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:42309 (local address 127.8.249.129:33031)
0926 20:52:52.279050 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:52.279066 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:52.279237 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:52.279745 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:52.279749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:52.279777 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:52.280123 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:52.280132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.281314 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.281318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:52.283574 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:52.283591 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.285297 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.285304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:52.285470 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:52.285501 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 20:52:52.286171 (+ 670us) client_negotiation.cc:240] Negotiation successful
0926 20:52:52.286323 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"mutex_wait_us":26,"thread_start_us":1128,"threads_started":1}
I20260926 20:52:52.287364 16561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4f9573726848598dbfbc4ec3028216" candidate_uuid: "db22faab4b5d414fadbce62b9eec0d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21ee59a5d9f4b7f867c0f978165a969" is_pre_election: true
I20260926 20:52:52.287505 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4f9573726848598dbfbc4ec3028216" candidate_uuid: "db22faab4b5d414fadbce62b9eec0d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00012a67b7e844bd8563f8cfaa0013b1" is_pre_election: true
I20260926 20:52:52.287892 16561 raft_consensus.cc:2454] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db22faab4b5d414fadbce62b9eec0d9e in term 0.
I20260926 20:52:52.287974 16420 raft_consensus.cc:2454] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db22faab4b5d414fadbce62b9eec0d9e in term 0.
I20260926 20:52:52.288556 16215 leader_election.cc:302] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db22faab4b5d414fadbce62b9eec0d9e, f21ee59a5d9f4b7f867c0f978165a969; no voters:
I20260926 20:52:52.288945 16649 raft_consensus.cc:2781] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:52:52.289090 16649 raft_consensus.cc:482] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:52.289199 16649 raft_consensus.cc:3037] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:52.291325 16649 raft_consensus.cc:504] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:52.291925 16649 leader_election.cc:288] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [CANDIDATE]: Term 1 election: Requested vote from peers f21ee59a5d9f4b7f867c0f978165a969 (127.8.249.131:39939), 00012a67b7e844bd8563f8cfaa0013b1 (127.8.249.130:42309)
I20260926 20:52:52.292304 16561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4f9573726848598dbfbc4ec3028216" candidate_uuid: "db22faab4b5d414fadbce62b9eec0d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21ee59a5d9f4b7f867c0f978165a969"
I20260926 20:52:52.292362 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4f9573726848598dbfbc4ec3028216" candidate_uuid: "db22faab4b5d414fadbce62b9eec0d9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00012a67b7e844bd8563f8cfaa0013b1"
I20260926 20:52:52.292511 16561 raft_consensus.cc:3037] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:52.292601 16420 raft_consensus.cc:3037] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:52.294852 16420 raft_consensus.cc:2454] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db22faab4b5d414fadbce62b9eec0d9e in term 1.
I20260926 20:52:52.294860 16561 raft_consensus.cc:2454] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db22faab4b5d414fadbce62b9eec0d9e in term 1.
I20260926 20:52:52.295257 16213 leader_election.cc:302] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00012a67b7e844bd8563f8cfaa0013b1, db22faab4b5d414fadbce62b9eec0d9e; no voters:
I20260926 20:52:52.295565 16649 raft_consensus.cc:2781] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:52.296005 16649 raft_consensus.cc:686] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [term 1 LEADER]: Becoming Leader. State: Replica: db22faab4b5d414fadbce62b9eec0d9e, State: Running, Role: LEADER
I20260926 20:52:52.296548 16649 consensus_queue.cc:237] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:52.301350 15349 catalog_manager.cc:5495] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e reported cstate change: term changed from 0 to 1, leader changed from <none> to db22faab4b5d414fadbce62b9eec0d9e (127.8.249.129). New cstate: current_term: 1 leader_uuid: "db22faab4b5d414fadbce62b9eec0d9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } health_report { overall_health: HEALTHY } } }
I20260926 20:52:52.356271 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:52.350957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.8.249.129:33315)
0926 20:52:52.351149 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:52.351153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:52.351172 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:52:52.351316 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:52.351319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:52.351393 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:52.351567 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:52.351957 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.352740 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.354264 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.354651 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.354740 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:52.355213 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:52.355713 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:52.355821 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:52.355911 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 20:52:52.355974 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:52:52.364316 16420 raft_consensus.cc:1261] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1 [term 1 FOLLOWER]: Refusing update from remote peer db22faab4b5d414fadbce62b9eec0d9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:52.364382 16561 raft_consensus.cc:1261] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969 [term 1 FOLLOWER]: Refusing update from remote peer db22faab4b5d414fadbce62b9eec0d9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:52.365303 16653 consensus_queue.cc:1035] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:52.365674 16649 consensus_queue.cc:1035] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:52.380518 16656 mvcc.cc:205] Tried to move back new op lower bound from 7333707662794878976 to 7333707662530154496. Current Snapshot: MvccSnapshot[applied={T|T < 7333707662794878976}]
I20260926 20:52:52.397543 15349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:52.397823 15349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:52.403532 15349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d6c916c8ec441e6be3173bb0f696d38: Sending DeleteTablet for 3 replicas of tablet 2b4f9573726848598dbfbc4ec3028216
I20260926 20:52:52.404646 16400 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4f9573726848598dbfbc4ec3028216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:52 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:51802
I20260926 20:52:52.404646 16541 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4f9573726848598dbfbc4ec3028216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:52 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:60922
I20260926 20:52:52.406852 16666 tablet_replica.cc:330] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1: stopping tablet replica
I20260926 20:52:52.407298 16666 raft_consensus.cc:2229] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:52.407789 16666 raft_consensus.cc:2258] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:52.408882 16667 tablet_replica.cc:330] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969: stopping tablet replica
I20260926 20:52:52.409065 16260 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4f9573726848598dbfbc4ec3028216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:52 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:41642
I20260926 20:52:52.409288 16667 raft_consensus.cc:2229] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:52.409725 16667 raft_consensus.cc:2258] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:52.409970 16668 tablet_replica.cc:330] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e: stopping tablet replica
I20260926 20:52:52.410647 16668 raft_consensus.cc:2229] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:52:52.411397 16668 raft_consensus.cc:2258] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:52.411723 16666 ts_tablet_manager.cc:1905] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:52.413769 16668 ts_tablet_manager.cc:1905] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:52.418824 16667 ts_tablet_manager.cc:1905] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:52.419106 16666 ts_tablet_manager.cc:1918] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:52.419260 16666 log.cc:1196] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/2b4f9573726848598dbfbc4ec3028216
I20260926 20:52:52.419806 16666 ts_tablet_manager.cc:1939] T 2b4f9573726848598dbfbc4ec3028216 P 00012a67b7e844bd8563f8cfaa0013b1: Deleting consensus metadata
I20260926 20:52:52.421567 15333 catalog_manager.cc:4841] TS 00012a67b7e844bd8563f8cfaa0013b1 (127.8.249.130:42309): tablet 2b4f9573726848598dbfbc4ec3028216 (table test-table [id=e123b77badc245e996389757189b6a15]) successfully deleted
I20260926 20:52:52.422101 16668 ts_tablet_manager.cc:1918] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:52.422297 16668 log.cc:1196] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/2b4f9573726848598dbfbc4ec3028216
I20260926 20:52:52.422955 16668 ts_tablet_manager.cc:1939] T 2b4f9573726848598dbfbc4ec3028216 P db22faab4b5d414fadbce62b9eec0d9e: Deleting consensus metadata
I20260926 20:52:52.425034 15336 catalog_manager.cc:4841] TS db22faab4b5d414fadbce62b9eec0d9e (127.8.249.129:33315): tablet 2b4f9573726848598dbfbc4ec3028216 (table test-table [id=e123b77badc245e996389757189b6a15]) successfully deleted
I20260926 20:52:52.426980 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 15317
I20260926 20:52:52.428066 15379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:52.432389 16667 ts_tablet_manager.cc:1918] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:52.432588 16667 log.cc:1196] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/2b4f9573726848598dbfbc4ec3028216
I20260926 20:52:52.433231 16667 ts_tablet_manager.cc:1939] T 2b4f9573726848598dbfbc4ec3028216 P f21ee59a5d9f4b7f867c0f978165a969: Deleting consensus metadata
I20260926 20:52:52.443809 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:52.411094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.8.249.189:46689)
0926 20:52:52.411289 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:52.411295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:52.425509 (+ 14214us) server_negotiation.cc:407] Connection header received
0926 20:52:52.425734 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:52.425739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:52.425821 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:52.425920 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:52.429096 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.430147 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.432393 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.433036 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.433149 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:52.436839 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:52.436849 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:52.436856 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:52.436915 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:52.440937 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:52.441585 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:52.441591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:52.441595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:52.441723 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:52.442104 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:52.442109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:52.442112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:52.442403 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:52.442656 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:52.443065 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 20:52:52.443306 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:52:52.457369 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:52.426269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.8.249.188:35777)
0926 20:52:52.426459 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:52.426465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:52.443671 (+ 17206us) server_negotiation.cc:407] Connection header received
0926 20:52:52.443830 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:52.443836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:52.443921 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:52.444019 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:52.445952 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.446703 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.448049 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.448544 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.448705 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:52.451547 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:52.451552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:52.451555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:52.451588 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:52.454705 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:52.455393 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:52.455400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:52.455404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:52.455534 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:52.455905 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:52.455911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:52.455914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:52.456233 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:52.456489 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:52.456795 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 20:52:52.457007 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:52:52.611267 15335 catalog_manager.cc:4841] TS f21ee59a5d9f4b7f867c0f978165a969 (127.8.249.131:39939): tablet 2b4f9573726848598dbfbc4ec3028216 (table test-table [id=e123b77badc245e996389757189b6a15]) successfully deleted
W20260926 20:52:52.612442 15335 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.249.131:39939, user_credentials={real_user=slave}} blocked reactor thread for 176764us
I20260926 20:52:52.613015 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 15317
I20260926 20:52:52.658497 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:52.640345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.8.249.189:46689)
0926 20:52:52.640529 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:52.640535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:52.641660 (+ 1125us) server_negotiation.cc:407] Connection header received
0926 20:52:52.641996 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:52.642005 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:52.642132 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:52.642227 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:52.644812 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.645925 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.647634 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.648174 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.648296 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:52.651998 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:52.652003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:52.652006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:52.652038 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:52.655367 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:52.656387 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:52.656393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:52.656396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:52.656526 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:52.657035 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:52.657040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:52.657043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:52.657332 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:52.657617 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:52.657934 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 20:52:52.658144 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:52:52.658739 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:52.641568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46308 (local address 127.8.249.188:35777)
0926 20:52:52.641754 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:52.641760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:52.642021 (+ 261us) server_negotiation.cc:407] Connection header received
0926 20:52:52.642270 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:52.642276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:52.642353 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:52.642422 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:52.644678 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.645487 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.647286 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:52.647978 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:52.648117 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:52.651949 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:52.651955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:52.651960 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:52.651998 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:52.655471 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:52.656393 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:52.656399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:52.656403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:52.656548 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:52.656996 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:52.657002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:52.657006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:52.657378 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:52.657610 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:52.658174 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 20:52:52.658409 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:52:53.388438 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.387723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.131:57919)
0926 20:52:53.388167 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:53.388279 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":245,"threads_started":1}
I20260926 20:52:53.388816 16679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.388090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.129:51361)
0926 20:52:53.388547 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:53.388678 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":250,"threads_started":1}
W20260926 20:52:53.388792 16607 heartbeater.cc:644] Failed to heartbeat to 127.8.249.190:44135 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
W20260926 20:52:53.389235 16326 heartbeater.cc:644] Failed to heartbeat to 127.8.249.190:44135 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
I20260926 20:52:53.392270 16680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.391555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.130:44053)
0926 20:52:53.391983 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:53.392077 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":240,"threads_started":1}
W20260926 20:52:53.392628 16466 heartbeater.cc:644] Failed to heartbeat to 127.8.249.190:44135 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
I20260926 20:52:53.904250 16681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9d6c916c8ec441e6be3173bb0f696d38)
I20260926 20:52:53.904563 16681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } }
I20260926 20:52:53.906066 16681 leader_election.cc:288] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135), aefc04b1a7444d8fbe73b7a2388d47d1 (127.8.249.189:46689)
I20260926 20:52:53.907229 16682 raft_consensus.cc:482] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9d6c916c8ec441e6be3173bb0f696d38)
I20260926 20:52:53.907441 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.906205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:49708)
0926 20:52:53.906819 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:53.907103 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":272,"threads_started":1}
I20260926 20:52:53.907426 16682 raft_consensus.cc:504] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } }
W20260926 20:52:53.907815 16131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
I20260926 20:52:53.909242 16682 leader_election.cc:288] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135), 0a04345301c847df90874e0f9cc23c84 (127.8.249.188:35777)
I20260926 20:52:53.909531 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.909059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:49730)
0926 20:52:53.909272 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:53.909359 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260926 20:52:53.910270 16131 leader_election.cc:334] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135): Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
I20260926 20:52:53.910866 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.908695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:49714)
0926 20:52:53.909898 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:53.910529 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1066,"thread_start_us":1171,"threads_started":1}
W20260926 20:52:53.911273 15941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
I20260926 20:52:53.912766 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.912439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:49732)
0926 20:52:53.912582 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:53.912663 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 20:52:53.913004 15941 leader_election.cc:334] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135): Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
I20260926 20:52:53.915560 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.906313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46689 (local address 127.0.0.1:47306)
0926 20:52:53.907371 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:53.907394 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:53.907580 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:53.908616 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:53.908620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:53.908644 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:53.908950 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:53.908965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:53.911637 (+ 2672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:53.911642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:53.913735 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:53.913750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:53.914928 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:53.914936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:53.915065 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:53.915093 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 20:52:53.915172 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:52:53.915271 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":725,"threads_started":1}
I20260926 20:52:53.915723 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.906470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.8.249.189:46689)
0926 20:52:53.907080 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:53.907088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:53.907500 (+ 412us) server_negotiation.cc:407] Connection header received
0926 20:52:53.908201 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:53.908210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:53.908321 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:53.908425 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:53.909158 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:53.911429 (+ 2271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:53.913871 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:53.914750 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:53.914922 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:53.915032 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:53.915219 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 20:52:53.915362 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":294,"threads_started":1}
I20260926 20:52:53.916416 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a04345301c847df90874e0f9cc23c84" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" is_pre_election: true
I20260926 20:52:53.916791 15964 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a04345301c847df90874e0f9cc23c84 in term 1.
I20260926 20:52:53.917321 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.909402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:35777 (local address 127.0.0.1:49544)
0926 20:52:53.910351 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:53.910374 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:53.910520 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:53.910866 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:53.910870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:53.910910 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:53.911231 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:53.911241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:53.912931 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:53.912945 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:53.914651 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:53.914662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:53.916416 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:53.916422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:53.916605 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:53.916640 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:52:53.916739 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 20:52:53.916891 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":223,"threads_started":1}
I20260926 20:52:53.917395 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.909551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.8.249.188:35777)
0926 20:52:53.910343 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:53.910352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:53.910556 (+ 204us) server_negotiation.cc:407] Connection header received
0926 20:52:53.910644 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:53.910649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:53.910714 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:53.910835 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:53.911434 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:53.912758 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:53.914854 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:53.916217 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:53.916318 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:53.916391 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:53.916768 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 20:52:53.916868 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":582,"threads_started":1}
I20260926 20:52:53.917446 16129 leader_election.cc:302] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a04345301c847df90874e0f9cc23c84, aefc04b1a7444d8fbe73b7a2388d47d1; no voters: 9d6c916c8ec441e6be3173bb0f696d38
I20260926 20:52:53.917975 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0a04345301c847df90874e0f9cc23c84" is_pre_election: true
I20260926 20:52:53.918329 16154 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aefc04b1a7444d8fbe73b7a2388d47d1 in term 1.
I20260926 20:52:53.918685 16681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 20:52:53.918818 16681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9d6c916c8ec441e6be3173bb0f696d38)
I20260926 20:52:53.918913 16681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:52:53.918834 15939 leader_election.cc:302] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a04345301c847df90874e0f9cc23c84, aefc04b1a7444d8fbe73b7a2388d47d1; no voters: 9d6c916c8ec441e6be3173bb0f696d38
I20260926 20:52:53.919214 16682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 20:52:53.919337 16682 raft_consensus.cc:482] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9d6c916c8ec441e6be3173bb0f696d38)
I20260926 20:52:53.919453 16682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:52:53.922036 16681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } }
I20260926 20:52:53.922394 16682 raft_consensus.cc:504] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } }
I20260926 20:52:53.922901 16681 leader_election.cc:288] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [CANDIDATE]: Term 2 election: Requested vote from peers 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135), aefc04b1a7444d8fbe73b7a2388d47d1 (127.8.249.189:46689)
I20260926 20:52:53.923514 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.923073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:49738)
0926 20:52:53.923269 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:53.923369 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 20:52:53.923507 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a04345301c847df90874e0f9cc23c84" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1"
I20260926 20:52:53.923652 16682 leader_election.cc:288] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [CANDIDATE]: Term 2 election: Requested vote from peers 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135), 0a04345301c847df90874e0f9cc23c84 (127.8.249.188:35777)
I20260926 20:52:53.923849 15964 raft_consensus.cc:2379] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a04345301c847df90874e0f9cc23c84 in current term 2: Already voted for candidate aefc04b1a7444d8fbe73b7a2388d47d1 in this term.
I20260926 20:52:53.923879 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.923227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:49750)
0926 20:52:53.923399 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:53.923501 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 20:52:53.924580 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0a04345301c847df90874e0f9cc23c84"
I20260926 20:52:53.924801 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.924172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:49756)
0926 20:52:53.924464 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:53.924567 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260926 20:52:53.924919 16154 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate aefc04b1a7444d8fbe73b7a2388d47d1 in current term 2: Already voted for candidate 0a04345301c847df90874e0f9cc23c84 in this term.
I20260926 20:52:53.925079 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:53.924462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:49772)
0926 20:52:53.924800 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:53.924895 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":184}
W20260926 20:52:53.925163 16131 leader_election.cc:334] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135): Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
W20260926 20:52:53.925231 15941 leader_election.cc:334] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135): Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
I20260926 20:52:53.925307 16131 leader_election.cc:302] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a04345301c847df90874e0f9cc23c84; no voters: 9d6c916c8ec441e6be3173bb0f696d38, aefc04b1a7444d8fbe73b7a2388d47d1
I20260926 20:52:53.925534 16681 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 20:52:53.925813 15939 leader_election.cc:302] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aefc04b1a7444d8fbe73b7a2388d47d1; no voters: 0a04345301c847df90874e0f9cc23c84, 9d6c916c8ec441e6be3173bb0f696d38
I20260926 20:52:53.926151 16682 raft_consensus.cc:2726] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 20:52:54.390152 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:54.389340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.131:49955)
0926 20:52:54.389809 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:54.389903 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":228,"threads_started":1}
I20260926 20:52:54.390561 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:54.389885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.129:60777)
0926 20:52:54.390280 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:54.390376 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":203,"threads_started":1}
I20260926 20:52:54.394039 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:54.393336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.130:60949)
0926 20:52:54.393787 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:54.393886 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":248,"threads_started":1}
I20260926 20:52:55.391850 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.391021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.131:47341)
0926 20:52:55.391533 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.391653 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":245,"threads_started":1}
I20260926 20:52:55.392105 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.391396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.129:60619)
0926 20:52:55.391777 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.391852 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":201,"threads_started":1}
I20260926 20:52:55.395897 16696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.394909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.130:43207)
0926 20:52:55.395634 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.395713 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":515,"threads_started":1}
I20260926 20:52:55.807871 16697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:55.808115 16697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } }
I20260926 20:52:55.808805 16697 leader_election.cc:288] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135), aefc04b1a7444d8fbe73b7a2388d47d1 (127.8.249.189:46689)
I20260926 20:52:55.809434 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a04345301c847df90874e0f9cc23c84" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" is_pre_election: true
I20260926 20:52:55.809691 15964 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a04345301c847df90874e0f9cc23c84 in term 2.
I20260926 20:52:55.809831 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.808962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:50054)
0926 20:52:55.809488 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.809603 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":220,"threads_started":1}
I20260926 20:52:55.810129 16129 leader_election.cc:302] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a04345301c847df90874e0f9cc23c84, aefc04b1a7444d8fbe73b7a2388d47d1; no voters:
I20260926 20:52:55.810441 16697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 20:52:55.810595 16697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:55.810700 16697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:52:55.811391 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.811077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:50064)
0926 20:52:55.811211 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.811274 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 20:52:55.811564 16131 leader_election.cc:334] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135): Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
I20260926 20:52:55.812968 16697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } }
I20260926 20:52:55.813477 16697 leader_election.cc:288] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [CANDIDATE]: Term 3 election: Requested vote from peers 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135), aefc04b1a7444d8fbe73b7a2388d47d1 (127.8.249.189:46689)
I20260926 20:52:55.813905 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.813604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:50080)
0926 20:52:55.813726 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.813796 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 20:52:55.813939 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a04345301c847df90874e0f9cc23c84" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1"
I20260926 20:52:55.814164 15964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:52:55.814865 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.814511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:50082)
0926 20:52:55.814655 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.814722 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 20:52:55.815002 16131 leader_election.cc:334] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135): Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
I20260926 20:52:55.816402 15964 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a04345301c847df90874e0f9cc23c84 in term 3.
I20260926 20:52:55.816834 16129 leader_election.cc:302] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a04345301c847df90874e0f9cc23c84, aefc04b1a7444d8fbe73b7a2388d47d1; no voters: 9d6c916c8ec441e6be3173bb0f696d38
I20260926 20:52:55.817130 16697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:52:55.817579 16697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [term 3 LEADER]: Becoming Leader. State: Replica: 0a04345301c847df90874e0f9cc23c84, State: Running, Role: LEADER
I20260926 20:52:55.817984 16697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } }
I20260926 20:52:55.819705 16700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a04345301c847df90874e0f9cc23c84. Latest consensus state: current_term: 3 leader_uuid: "0a04345301c847df90874e0f9cc23c84" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } } }
I20260926 20:52:55.820047 16700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:55.820700 16702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:55.824342 16702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:55.825174 16702 catalog_manager.cc:1259] Loaded cluster ID: 986a280ad246442aafa35051eae5c0c7
I20260926 20:52:55.825276 16702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:55.826395 16702 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:55.827227 16702 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: Loaded TSK: 0
I20260926 20:52:55.828087 16702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:52:55.851158 16143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:52:55.858306 16143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:52:55.862731 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.862219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:50106)
0926 20:52:55.862452 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.862551 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 20:52:55.862697 15964 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [term 3 FOLLOWER]: Refusing update from remote peer 0a04345301c847df90874e0f9cc23c84: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 20:52:55.863490 16700 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 20:52:55.863744 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.863404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:50122)
0926 20:52:55.863550 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.863618 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 20:52:55.863926 16131 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 -> Peer 9d6c916c8ec441e6be3173bb0f696d38 (127.8.249.190:44135): Couldn't send request to peer 9d6c916c8ec441e6be3173bb0f696d38. Status: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 20:52:55.865896 16704 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a04345301c847df90874e0f9cc23c84. Latest consensus state: current_term: 3 leader_uuid: "0a04345301c847df90874e0f9cc23c84" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } } }
I20260926 20:52:55.866127 16704 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:52:55.868201 16697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0a04345301c847df90874e0f9cc23c84" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } } }
I20260926 20:52:55.868629 16697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:55.868631 16704 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0a04345301c847df90874e0f9cc23c84" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d6c916c8ec441e6be3173bb0f696d38" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44135 } } peers { permanent_uuid: "aefc04b1a7444d8fbe73b7a2388d47d1" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46689 } attrs { promote: false } } peers { permanent_uuid: "0a04345301c847df90874e0f9cc23c84" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 35777 } attrs { promote: false } } }
I20260926 20:52:55.868892 16704 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefc04b1a7444d8fbe73b7a2388d47d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:52:55.869331 16705 mvcc.cc:205] Tried to move back new op lower bound from 7333707677126844416 to 7333707676954439680. Current Snapshot: MvccSnapshot[applied={T|T < 7333707677126844416 or (T in {7333707677126844416})}]
I20260926 20:52:55.894672 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.877875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:42309 (local address 127.0.0.1:59634)
0926 20:52:55.878496 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.878513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:55.878624 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:55.887534 (+ 8910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:55.887538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:55.887554 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:55.888247 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:55.888260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.889817 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.889825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:55.891886 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:55.891897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.893747 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.893755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:55.893923 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:55.893955 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 20:52:55.894044 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:52:55.894205 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":80,"thread_start_us":1677,"threads_started":1}
I20260926 20:52:55.895392 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.879254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.8.249.129:33315)
0926 20:52:55.886996 (+ 7742us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:55.887006 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:55.887038 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:52:55.887152 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:55.887160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:55.887272 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:55.887430 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:55.888766 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.889808 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.892419 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.893787 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.893887 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:55.893989 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:55.894576 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 20:52:55.894750 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7546,"thread_start_us":7373,"threads_started":1}
I20260926 20:52:55.895411 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.878055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.8.249.130:42309)
0926 20:52:55.887208 (+ 9153us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:55.887212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:55.887232 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:52:55.887291 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:55.887294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:55.887359 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:55.887662 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:55.888477 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.889641 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.892052 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.893364 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.893513 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:55.893635 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:55.894946 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0926 20:52:55.895086 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9025,"thread_start_us":9553,"threads_started":1}
I20260926 20:52:55.895088 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.878858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:33315 (local address 127.0.0.1:43742)
0926 20:52:55.879329 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.879347 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:55.879463 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:55.887489 (+ 8026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:55.887496 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:55.887532 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:55.888562 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:55.888577 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.889972 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.889978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:55.892181 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:55.892196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.894177 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.894183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:55.894346 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:55.894375 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 20:52:55.894516 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 20:52:55.894660 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":278,"thread_start_us":596,"threads_started":1}
I20260926 20:52:55.896585 16260 tablet_service.cc:1467] Processing CreateTablet for tablet 18d66a4a4b104f67a2718cb64aafda1b (DEFAULT_TABLE table=test-table [id=5cd3e42b6b6a42f7ad7097de77db2fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:55.896899 16400 tablet_service.cc:1467] Processing CreateTablet for tablet 18d66a4a4b104f67a2718cb64aafda1b (DEFAULT_TABLE table=test-table [id=5cd3e42b6b6a42f7ad7097de77db2fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:55.897058 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.877468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:39939 (local address 127.0.0.1:50878)
0926 20:52:55.877639 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.877660 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:55.877777 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:55.890735 (+ 12958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:55.890741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:55.890770 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:55.891194 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:55.891208 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.892598 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.892603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:55.894565 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:55.894576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.896396 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.896402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:55.896548 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:55.896585 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:52:55.896680 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:52:55.896807 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 20:52:55.897097 16260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18d66a4a4b104f67a2718cb64aafda1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:55.897367 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18d66a4a4b104f67a2718cb64aafda1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:55.897805 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.877642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.8.249.131:39939)
0926 20:52:55.890361 (+ 12719us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:55.890367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:55.890386 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:52:55.890472 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:55.890477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:55.890563 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:55.890676 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:55.891374 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.892449 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.894756 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.896154 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.896330 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:55.896430 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:55.897363 (+ 933us) server_negotiation.cc:299] Negotiation successful
0926 20:52:55.897518 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12571,"thread_start_us":12449,"threads_started":1}
I20260926 20:52:55.902709 16714 tablet_bootstrap.cc:492] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e: Bootstrap starting.
I20260926 20:52:55.902863 16541 tablet_service.cc:1467] Processing CreateTablet for tablet 18d66a4a4b104f67a2718cb64aafda1b (DEFAULT_TABLE table=test-table [id=5cd3e42b6b6a42f7ad7097de77db2fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:52:55.903369 16541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18d66a4a4b104f67a2718cb64aafda1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:52:55.905427 16714 tablet_bootstrap.cc:654] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:55.907083 16715 tablet_bootstrap.cc:492] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1: Bootstrap starting.
I20260926 20:52:55.908850 16714 tablet_bootstrap.cc:492] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e: No bootstrap required, opened a new log
I20260926 20:52:55.909009 16714 ts_tablet_manager.cc:1397] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 20:52:55.909132 16715 tablet_bootstrap.cc:654] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:55.909931 16714 raft_consensus.cc:348] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:55.910537 16714 raft_consensus.cc:374] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:55.910647 16717 tablet_bootstrap.cc:492] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969: Bootstrap starting.
I20260926 20:52:55.910696 16714 raft_consensus.cc:729] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db22faab4b5d414fadbce62b9eec0d9e, State: Initialized, Role: FOLLOWER
I20260926 20:52:55.911044 16714 consensus_queue.cc:260] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:55.911763 16715 tablet_bootstrap.cc:492] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1: No bootstrap required, opened a new log
I20260926 20:52:55.911912 16715 ts_tablet_manager.cc:1397] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:52:55.912271 16714 ts_tablet_manager.cc:1428] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 20:52:55.912863 16715 raft_consensus.cc:348] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:55.913223 16715 raft_consensus.cc:374] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:55.913343 16715 raft_consensus.cc:729] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00012a67b7e844bd8563f8cfaa0013b1, State: Initialized, Role: FOLLOWER
I20260926 20:52:55.913580 16717 tablet_bootstrap.cc:654] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:55.914433 16715 consensus_queue.cc:260] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:55.917059 16325 heartbeater.cc:503] Master 127.8.249.188:35777 was elected leader, sending a full tablet report...
I20260926 20:52:55.921641 16715 ts_tablet_manager.cc:1428] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260926 20:52:55.922086 16717 tablet_bootstrap.cc:492] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969: No bootstrap required, opened a new log
I20260926 20:52:55.922180 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.912377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.129:33209)
0926 20:52:55.921750 (+ 9373us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.921895 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9240,"thread_start_us":9112,"threads_started":1}
I20260926 20:52:55.922268 16717 ts_tablet_manager.cc:1397] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260926 20:52:55.923240 16717 raft_consensus.cc:348] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:55.923561 16717 raft_consensus.cc:374] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:55.923722 16717 raft_consensus.cc:729] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21ee59a5d9f4b7f867c0f978165a969, State: Initialized, Role: FOLLOWER
I20260926 20:52:55.924057 16717 consensus_queue.cc:260] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:55.924669 16465 heartbeater.cc:503] Master 127.8.249.188:35777 was elected leader, sending a full tablet report...
I20260926 20:52:55.929476 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.928840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.129:45843)
0926 20:52:55.929113 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.929252 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 20:52:55.930428 16717 ts_tablet_manager.cc:1428] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260926 20:52:55.931790 16723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.923014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.130:40969)
0926 20:52:55.930976 (+ 7962us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.931512 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7772,"thread_start_us":7325,"threads_started":1}
I20260926 20:52:55.933017 16606 heartbeater.cc:503] Master 127.8.249.188:35777 was elected leader, sending a full tablet report...
I20260926 20:52:55.933068 16725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.931405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.131:54539)
0926 20:52:55.932619 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.932778 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":1251,"threads_started":1}
I20260926 20:52:55.933444 16723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.932651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.130:51513)
0926 20:52:55.933222 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.933306 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465}
I20260926 20:52:55.934430 16725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.934022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.131:56229)
0926 20:52:55.934195 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.934279 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 20:52:55.977800 16722 raft_consensus.cc:482] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:52:55.978101 16722 raft_consensus.cc:504] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:55.979918 16722 leader_election.cc:288] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21ee59a5d9f4b7f867c0f978165a969 (127.8.249.131:39939), db22faab4b5d414fadbce62b9eec0d9e (127.8.249.129:33315)
I20260926 20:52:55.986665 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.980377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:55955 (local address 127.8.249.129:33315)
0926 20:52:55.980584 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:55.980590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:55.980712 (+ 122us) server_negotiation.cc:407] Connection header received
0926 20:52:55.980958 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:55.980966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:55.981067 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:55.981163 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:55.981858 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.983005 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.984836 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.985637 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.985835 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:55.985925 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:55.986326 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 20:52:55.986461 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:52:55.987224 16723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.980297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:33315 (local address 127.8.249.130:55955)
0926 20:52:55.980580 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.980602 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:55.980806 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:55.981321 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:55.981328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:55.981361 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:55.981700 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:55.981713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.983124 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.983129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:55.984547 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:55.984560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.985832 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.985839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:55.986121 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:55.986169 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 20:52:55.986892 (+ 723us) client_negotiation.cc:240] Negotiation successful
0926 20:52:55.987019 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":3}
I20260926 20:52:55.987232 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.980394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:45165 (local address 127.8.249.131:39939)
0926 20:52:55.980580 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:55.980587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:55.981468 (+ 881us) server_negotiation.cc:407] Connection header received
0926 20:52:55.981736 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:55.981747 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:55.981841 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:55.982079 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:52:55.982510 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.983421 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.985180 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.986065 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.986222 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:55.986292 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:55.986897 (+ 605us) server_negotiation.cc:299] Negotiation successful
0926 20:52:55.987046 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:52:55.987943 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.980300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:39939 (local address 127.8.249.130:45165)
0926 20:52:55.981370 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.981395 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:55.981587 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:55.982088 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:55.982091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:55.982112 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:52:55.982363 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:55.982371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.983560 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.983570 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:55.985013 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:55.985026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:55.986461 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:55.986470 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:55.986663 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:55.986706 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:52:55.987448 (+ 742us) client_negotiation.cc:240] Negotiation successful
0926 20:52:55.987589 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":257,"thread_start_us":500,"threads_started":1}
I20260926 20:52:55.988176 16280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18d66a4a4b104f67a2718cb64aafda1b" candidate_uuid: "00012a67b7e844bd8563f8cfaa0013b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db22faab4b5d414fadbce62b9eec0d9e" is_pre_election: true
I20260926 20:52:55.988581 16561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18d66a4a4b104f67a2718cb64aafda1b" candidate_uuid: "00012a67b7e844bd8563f8cfaa0013b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21ee59a5d9f4b7f867c0f978165a969" is_pre_election: true
I20260926 20:52:55.988629 16280 raft_consensus.cc:2454] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00012a67b7e844bd8563f8cfaa0013b1 in term 0.
I20260926 20:52:55.988819 16561 raft_consensus.cc:2454] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00012a67b7e844bd8563f8cfaa0013b1 in term 0.
I20260926 20:52:55.989229 16356 leader_election.cc:302] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00012a67b7e844bd8563f8cfaa0013b1, db22faab4b5d414fadbce62b9eec0d9e; no voters:
I20260926 20:52:55.989641 16722 raft_consensus.cc:2781] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:52:55.989779 16722 raft_consensus.cc:482] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:52:55.989881 16722 raft_consensus.cc:3037] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:55.992096 16722 raft_consensus.cc:504] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:55.992682 16722 leader_election.cc:288] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [CANDIDATE]: Term 1 election: Requested vote from peers f21ee59a5d9f4b7f867c0f978165a969 (127.8.249.131:39939), db22faab4b5d414fadbce62b9eec0d9e (127.8.249.129:33315)
I20260926 20:52:55.993256 16561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18d66a4a4b104f67a2718cb64aafda1b" candidate_uuid: "00012a67b7e844bd8563f8cfaa0013b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21ee59a5d9f4b7f867c0f978165a969"
I20260926 20:52:55.993256 16280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18d66a4a4b104f67a2718cb64aafda1b" candidate_uuid: "00012a67b7e844bd8563f8cfaa0013b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db22faab4b5d414fadbce62b9eec0d9e"
I20260926 20:52:55.993498 16561 raft_consensus.cc:3037] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:55.993515 16280 raft_consensus.cc:3037] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:55.996877 16561 raft_consensus.cc:2454] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00012a67b7e844bd8563f8cfaa0013b1 in term 1.
I20260926 20:52:55.996877 16280 raft_consensus.cc:2454] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00012a67b7e844bd8563f8cfaa0013b1 in term 1.
I20260926 20:52:55.997426 16356 leader_election.cc:302] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00012a67b7e844bd8563f8cfaa0013b1, db22faab4b5d414fadbce62b9eec0d9e, f21ee59a5d9f4b7f867c0f978165a969; no voters:
I20260926 20:52:55.997830 16722 raft_consensus.cc:2781] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:55.998313 16722 raft_consensus.cc:686] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [term 1 LEADER]: Becoming Leader. State: Replica: 00012a67b7e844bd8563f8cfaa0013b1, State: Running, Role: LEADER
I20260926 20:52:55.998739 16722 consensus_queue.cc:237] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } }
I20260926 20:52:55.999742 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:55.999281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.130:40571)
0926 20:52:55.999459 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:55.999571 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:52:56.002529 16144 catalog_manager.cc:5495] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00012a67b7e844bd8563f8cfaa0013b1 (127.8.249.130). New cstate: current_term: 1 leader_uuid: "00012a67b7e844bd8563f8cfaa0013b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00012a67b7e844bd8563f8cfaa0013b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 42309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 } health_report { overall_health: UNKNOWN } } }
I20260926 20:52:56.047227 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:56.042408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.8.249.130:42309)
0926 20:52:56.042597 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:56.042602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:56.042620 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:52:56.042768 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:56.042772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:56.042917 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:56.043048 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:52:56.043338 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:56.044038 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:56.045207 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:56.045583 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:56.045669 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:56.046026 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:52:56.046563 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:52:56.046675 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:56.046741 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 20:52:56.046869 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:52:56.051627 16561 raft_consensus.cc:1261] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969 [term 1 FOLLOWER]: Refusing update from remote peer 00012a67b7e844bd8563f8cfaa0013b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:56.051928 16280 raft_consensus.cc:1261] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e [term 1 FOLLOWER]: Refusing update from remote peer 00012a67b7e844bd8563f8cfaa0013b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:52:56.052457 16727 consensus_queue.cc:1035] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21ee59a5d9f4b7f867c0f978165a969" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:56.052891 16722 consensus_queue.cc:1035] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db22faab4b5d414fadbce62b9eec0d9e" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:52:56.055826 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:56.055272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.129:53029)
0926 20:52:56.055521 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:56.055635 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260926 20:52:56.055842 16725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:56.055367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.131:47745)
0926 20:52:56.055542 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:56.055681 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 20:52:56.064488 16731 mvcc.cc:205] Tried to move back new op lower bound from 7333707677900087296 to 7333707677695094784. Current Snapshot: MvccSnapshot[applied={T|T < 7333707677900087296}]
I20260926 20:52:56.066342 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:56.065822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.129:52443)
0926 20:52:56.066051 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:56.066152 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260926 20:52:56.069312 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:56.068826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.130:55951)
0926 20:52:56.069014 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:56.069127 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 20:52:56.069464 16732 mvcc.cc:205] Tried to move back new op lower bound from 7333707677900087296 to 7333707677695094784. Current Snapshot: MvccSnapshot[applied={T|T < 7333707677900087296}]
I20260926 20:52:56.071210 16725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:56.070470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.8.249.131:50163)
0926 20:52:56.070892 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:56.071009 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247}
I20260926 20:52:56.081344 16143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:56.081648 16143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:52:56.087519 16143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a04345301c847df90874e0f9cc23c84: Sending DeleteTablet for 3 replicas of tablet 18d66a4a4b104f67a2718cb64aafda1b
I20260926 20:52:56.088730 16541 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d66a4a4b104f67a2718cb64aafda1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:56 UTC) from {username='oryx', principal='oryx/127.8.249.188@KRBTEST.COM'} at 127.0.0.1:50878
I20260926 20:52:56.088785 16400 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d66a4a4b104f67a2718cb64aafda1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:56 UTC) from {username='oryx', principal='oryx/127.8.249.188@KRBTEST.COM'} at 127.0.0.1:59634
I20260926 20:52:56.090284 16260 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d66a4a4b104f67a2718cb64aafda1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:52:56 UTC) from {username='oryx', principal='oryx/127.8.249.188@KRBTEST.COM'} at 127.0.0.1:43742
I20260926 20:52:56.091144 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 16122
I20260926 20:52:56.094339 16741 tablet_replica.cc:330] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969: stopping tablet replica
I20260926 20:52:56.094621 16741 raft_consensus.cc:2229] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:56.095067 16741 raft_consensus.cc:2258] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:56.096361 16741 ts_tablet_manager.cc:1905] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:56.102837 16742 tablet_replica.cc:330] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e: stopping tablet replica
I20260926 20:52:56.103169 16742 raft_consensus.cc:2229] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:52:56.103637 16742 raft_consensus.cc:2258] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:56.105334 16743 tablet_replica.cc:330] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1: stopping tablet replica
I20260926 20:52:56.105794 16742 ts_tablet_manager.cc:1905] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:56.106138 16743 raft_consensus.cc:2229] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:52:56.106769 16743 raft_consensus.cc:2258] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:52:56.108400 16741 ts_tablet_manager.cc:1918] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:56.108495 16743 ts_tablet_manager.cc:1905] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:52:56.108592 16741 log.cc:1196] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/18d66a4a4b104f67a2718cb64aafda1b
I20260926 20:52:56.109153 16741 ts_tablet_manager.cc:1939] T 18d66a4a4b104f67a2718cb64aafda1b P f21ee59a5d9f4b7f867c0f978165a969: Deleting consensus metadata
I20260926 20:52:56.110427 16130 catalog_manager.cc:4841] TS f21ee59a5d9f4b7f867c0f978165a969 (127.8.249.131:39939): tablet 18d66a4a4b104f67a2718cb64aafda1b (table test-table [id=5cd3e42b6b6a42f7ad7097de77db2fdb]) successfully deleted
I20260926 20:52:56.114890 16742 ts_tablet_manager.cc:1918] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:56.115074 16742 log.cc:1196] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/18d66a4a4b104f67a2718cb64aafda1b
I20260926 20:52:56.115577 16742 ts_tablet_manager.cc:1939] T 18d66a4a4b104f67a2718cb64aafda1b P db22faab4b5d414fadbce62b9eec0d9e: Deleting consensus metadata
I20260926 20:52:56.116791 16131 catalog_manager.cc:4841] TS db22faab4b5d414fadbce62b9eec0d9e (127.8.249.129:33315): tablet 18d66a4a4b104f67a2718cb64aafda1b (table test-table [id=5cd3e42b6b6a42f7ad7097de77db2fdb]) successfully deleted
I20260926 20:52:56.117980 16743 ts_tablet_manager.cc:1918] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:52:56.118163 16743 log.cc:1196] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/18d66a4a4b104f67a2718cb64aafda1b
I20260926 20:52:56.118613 16743 ts_tablet_manager.cc:1939] T 18d66a4a4b104f67a2718cb64aafda1b P 00012a67b7e844bd8563f8cfaa0013b1: Deleting consensus metadata
I20260926 20:52:56.119638 16128 catalog_manager.cc:4841] TS 00012a67b7e844bd8563f8cfaa0013b1 (127.8.249.130:42309): tablet 18d66a4a4b104f67a2718cb64aafda1b (table test-table [id=5cd3e42b6b6a42f7ad7097de77db2fdb]) successfully deleted
I20260926 20:52:56.124094 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:56.093724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.8.249.129:33315)
0926 20:52:56.093919 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:56.093926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:56.105686 (+ 11760us) server_negotiation.cc:407] Connection header received
0926 20:52:56.105875 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:56.105882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:56.105966 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:56.106095 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:56.109572 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:56.110617 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:56.112468 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:56.113145 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:56.113295 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:56.117079 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:56.117098 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:56.117119 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:56.117166 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:56.121125 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:56.122043 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:56.122050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:56.122052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:56.122202 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:56.122643 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:56.122647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:56.122649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:56.123059 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:56.123362 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:56.123648 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 20:52:56.123811 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:52:56.124809 16320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:56.256232 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:56.255698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:50126)
0926 20:52:56.255909 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:56.256052 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 20:52:56.257467 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:56.257059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44135 (local address 127.0.0.1:50138)
0926 20:52:56.257221 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:56.257310 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 20:52:56.261374 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 16122
I20260926 20:52:56.283769 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 16337
I20260926 20:52:56.298187 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:56.284519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.8.249.130:42309)
0926 20:52:56.284713 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:56.284720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:56.284915 (+ 195us) server_negotiation.cc:407] Connection header received
0926 20:52:56.285121 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:56.285130 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:56.285227 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:56.285346 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:56.286618 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:56.287660 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:56.288927 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:56.289366 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:56.289515 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:56.292084 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:56.292103 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:56.292120 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:56.292158 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:56.295368 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:56.296082 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:56.296090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:56.296092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:56.296264 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:56.296626 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:56.296632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:56.296637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:56.296974 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:56.297304 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:56.297525 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 20:52:56.297748 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:52:56.298967 16460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:56.418267 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 16337
I20260926 20:52:56.437901 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 16478
I20260926 20:52:56.451278 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:56.438683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.8.249.131:39939)
0926 20:52:56.438907 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:56.438911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:56.438930 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:52:56.439072 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:56.439076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:56.439146 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:56.439228 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:56.440505 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:56.441219 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:56.442332 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:56.442707 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:56.442855 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:56.445165 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:56.445180 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:56.445196 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:56.445229 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:56.448294 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:56.448928 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:56.448936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:56.448938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:56.449074 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:56.449363 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:56.449366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:56.449368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:56.449710 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:56.450051 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:56.450371 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 20:52:56.450606 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:52:56.452055 16601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:56.545578 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 16478
I20260926 20:52:56.563460 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 15810
I20260926 20:52:56.564607 15984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:56.684965 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 15810
I20260926 20:52:56.708554 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 15911
I20260926 20:52:56.709599 16174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:52:56.828925 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 15911
2026-09-26T20:52:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11622 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:52:56 dist-test-slave-2dg3 krb5kdc[16752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:52:56 dist-test-slave-2dg3 krb5kdc[16752](info): set up 2 sockets
Sep 26 20:52:56 dist-test-slave-2dg3 krb5kdc[16752](info): commencing operation
krb5kdc: starting...
W20260926 20:52:57.928113 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:52:58 dist-test-slave-2dg3 krb5kdc[16752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:52:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:52:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.190.keytab.
Entry for principal oryx/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.190.keytab.
I20260926 20:52:58.142468 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:40349
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.190.keytab
--principal=oryx/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:40349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:58.462137 16765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:58.462410 16765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:58.462577 16765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:58.469822 16765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:52:58.469926 16765 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:52:58.469967 16765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:58.470098 16765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:58.470175 16765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:52:58.470279 16765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:52:58.480121 16765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:40349
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.190.keytab
--principal=oryx/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:40349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:58.481417 16765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:58.482770 16765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:58.494783 16771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:58.494946 16770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:58.494783 16773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:58.495441 16765 server_base.cc:1034] running on GCE node
I20260926 20:52:58.496289 16765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:58.497776 16765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:58.499015 16765 hybrid_clock.cc:648] HybridClock initialized: now 1790455978498939 us; error 64 us; skew 500 ppm
I20260926 20:52:58.499360 16765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:58 dist-test-slave-2dg3 krb5kdc[16752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455978, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:58.502480 16765 init.cc:377] Logged in from keytab as oryx/127.8.249.190@KRBTEST.COM (short username oryx)
I20260926 20:52:58.504106 16765 webserver.cc:460] Webserver started at http://127.8.249.190:37083/ using document root <none> and password file <none>
I20260926 20:52:58.504861 16765 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:58.504993 16765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:58.505333 16765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:58.508404 16765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "b221558242e242b6af9bf7d678e94585"
format_stamp: "Formatted at 2026-09-26 20:52:58 on dist-test-slave-2dg3"
server_key: "883a0034f59e3e1c1e2e02747cfb8203"
server_key_iv: "9245f16a5fc40e4d50b080afb4a3473d"
server_key_version: "encryptionkey@0"
I20260926 20:52:58.509214 16765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "b221558242e242b6af9bf7d678e94585"
format_stamp: "Formatted at 2026-09-26 20:52:58 on dist-test-slave-2dg3"
server_key: "883a0034f59e3e1c1e2e02747cfb8203"
server_key_iv: "9245f16a5fc40e4d50b080afb4a3473d"
server_key_version: "encryptionkey@0"
I20260926 20:52:58.514382 16765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260926 20:52:58.517899 16780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:58.519306 16765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:52:58.519487 16765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "b221558242e242b6af9bf7d678e94585"
format_stamp: "Formatted at 2026-09-26 20:52:58 on dist-test-slave-2dg3"
server_key: "883a0034f59e3e1c1e2e02747cfb8203"
server_key_iv: "9245f16a5fc40e4d50b080afb4a3473d"
server_key_version: "encryptionkey@0"
I20260926 20:52:58.519711 16765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:58.555549 16765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:58.559731 16765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:58.560014 16765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:58.581319 16765 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:40349
I20260926 20:52:58.581326 16841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:40349 every 8 connection(s)
I20260926 20:52:58.582741 16765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:52:58.583690 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 16765
I20260926 20:52:58.583911 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:52:58.584700 9190 external_mini_cluster.cc:1428] Setting key a2102a1edfb414363404285e56d1a829
I20260926 20:52:58.587715 16842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 20:52:58 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:52:58.601089 16842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585: Bootstrap starting.
I20260926 20:52:58.605614 16842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585: Neither blocks nor log segments found. Creating new log.
I20260926 20:52:58.606901 16842 log.cc:824] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585: Log is configured to *not* fsync() on all Append() calls
I20260926 20:52:58.610414 16842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585: No bootstrap required, opened a new log
I20260926 20:52:58.610574 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:58.587036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.8.249.190:40349)
0926 20:52:58.587981 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:58.587994 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:58.588045 (+ 51us) server_negotiation.cc:407] Connection header received
0926 20:52:58.589805 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:58.589831 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:58.590293 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:58.590699 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:58.592564 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:58.593983 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:58.595779 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:58.596491 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:58.596649 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:58.601529 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:58.601541 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:58.601563 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:58.601615 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:58.605727 (+ 4112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:58.606522 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:58.606534 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:58.606545 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:58.606765 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:58.607292 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:58.607297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:58.607301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:58.608008 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:58.608245 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:58.608790 (+ 545us) server_negotiation.cc:299] Negotiation successful
0926 20:52:58.609229 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":434,"threads_started":1}
I20260926 20:52:58.618376 16842 raft_consensus.cc:348] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b221558242e242b6af9bf7d678e94585" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40349 } }
I20260926 20:52:58.618824 16842 raft_consensus.cc:374] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:52:58.618975 16842 raft_consensus.cc:729] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b221558242e242b6af9bf7d678e94585, State: Initialized, Role: FOLLOWER
I20260926 20:52:58.619629 16842 consensus_queue.cc:260] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b221558242e242b6af9bf7d678e94585" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40349 } }
I20260926 20:52:58.619886 16842 raft_consensus.cc:388] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:52:58.620018 16842 raft_consensus.cc:482] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:52:58.620205 16842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:52:58.622334 16842 raft_consensus.cc:504] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b221558242e242b6af9bf7d678e94585" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40349 } }
I20260926 20:52:58.622929 16842 leader_election.cc:302] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b221558242e242b6af9bf7d678e94585; no voters:
I20260926 20:52:58.623553 16842 leader_election.cc:288] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:52:58.623641 16847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:52:58.624426 16847 raft_consensus.cc:686] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [term 1 LEADER]: Becoming Leader. State: Replica: b221558242e242b6af9bf7d678e94585, State: Running, Role: LEADER
I20260926 20:52:58.624969 16847 consensus_queue.cc:237] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b221558242e242b6af9bf7d678e94585" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40349 } }
I20260926 20:52:58.625685 16842 sys_catalog.cc:564] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:52:58.627602 16848 sys_catalog.cc:455] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b221558242e242b6af9bf7d678e94585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b221558242e242b6af9bf7d678e94585" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40349 } } }
I20260926 20:52:58.627682 16849 sys_catalog.cc:455] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b221558242e242b6af9bf7d678e94585. Latest consensus state: current_term: 1 leader_uuid: "b221558242e242b6af9bf7d678e94585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b221558242e242b6af9bf7d678e94585" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40349 } } }
I20260926 20:52:58.628161 16849 sys_catalog.cc:458] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:58.628161 16848 sys_catalog.cc:458] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585 [sys.catalog]: This master's current role is: LEADER
I20260926 20:52:58.630538 16854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:52:58.636083 16854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:52:58.646133 16854 catalog_manager.cc:1347] Generated new cluster ID: 6c4e6e5792874ae4b985be94f92fdb44
I20260926 20:52:58.646266 16854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:52:58.676276 16854 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:52:58.677690 16854 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:52:58.689648 16854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585: Generated new TSK 0
I20260926 20:52:58.690629 16854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab.
Entry for principal oryx/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab.
I20260926 20:52:58.759775 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40349
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab
--principal=oryx/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:59.121307 16868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:59.121675 16868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:59.121845 16868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:59.129827 16868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:59.129992 16868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:59.130162 16868 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:52:59.130306 16868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:52:59.141057 16868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab
--principal=oryx/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.249.190:40349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:59.142441 16868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:59.143915 16868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:59.157363 16873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:59.157227 16874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:59.157312 16876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:59.158310 16868 server_base.cc:1034] running on GCE node
I20260926 20:52:59.159024 16868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:59.159940 16868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:59.161188 16868 hybrid_clock.cc:648] HybridClock initialized: now 1790455979161119 us; error 41 us; skew 500 ppm
I20260926 20:52:59.161496 16868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:59 dist-test-slave-2dg3 krb5kdc[16752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455979, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:59.164870 16868 init.cc:377] Logged in from keytab as oryx/127.8.249.129@KRBTEST.COM (short username oryx)
I20260926 20:52:59.166405 16868 webserver.cc:460] Webserver started at http://127.8.249.129:43723/ using document root <none> and password file <none>
I20260926 20:52:59.167184 16868 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:59.167309 16868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:59.167629 16868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:59.170557 16868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd"
format_stamp: "Formatted at 2026-09-26 20:52:59 on dist-test-slave-2dg3"
server_key: "2dd56f42828dced4d14fdedcf600922c"
server_key_iv: "4b6bc16099341d1edc4eb0a305729f12"
server_key_version: "encryptionkey@0"
I20260926 20:52:59.171453 16868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd"
format_stamp: "Formatted at 2026-09-26 20:52:59 on dist-test-slave-2dg3"
server_key: "2dd56f42828dced4d14fdedcf600922c"
server_key_iv: "4b6bc16099341d1edc4eb0a305729f12"
server_key_version: "encryptionkey@0"
I20260926 20:52:59.176594 16868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:52:59.180001 16883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:59.181308 16868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 20:52:59.181484 16868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd"
format_stamp: "Formatted at 2026-09-26 20:52:59 on dist-test-slave-2dg3"
server_key: "2dd56f42828dced4d14fdedcf600922c"
server_key_iv: "4b6bc16099341d1edc4eb0a305729f12"
server_key_version: "encryptionkey@0"
I20260926 20:52:59.181707 16868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:59.205380 16868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:59.210464 16868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:59.210847 16868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:59.214363 16868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:59.214470 16868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:59.214598 16868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:59.214669 16868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:52:59 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455979, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:52:59.246520 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:59.215461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:48657 (local address 127.8.249.190:40349)
0926 20:52:59.215951 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:59.215958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:59.217952 (+ 1994us) server_negotiation.cc:407] Connection header received
0926 20:52:59.221587 (+ 3635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:59.221597 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:59.221728 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:59.221828 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:59.225428 (+ 3600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.226950 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.230884 (+ 3934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.231766 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.231928 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:59.238033 (+ 6105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:59.238043 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:59.238051 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:59.238111 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:59.242488 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:59.243781 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:59.243789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:59.243794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:59.243966 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:59.244659 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:59.244663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:59.244665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:59.244928 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:59.245092 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:59.245881 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 20:52:59.246137 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":235,"threads_started":1}
I20260926 20:52:59.248179 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:59.216815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40349 (local address 127.8.249.129:48657)
0926 20:52:59.217755 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:59.217806 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:59.221126 (+ 3320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:59.222562 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:59.222580 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:59.223696 (+ 1116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:59.225093 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:59.225139 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.227291 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.227300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:59.230605 (+ 3305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:59.230637 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.232256 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.232270 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:59.232527 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:59.233384 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:59.233424 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:59.237600 (+ 4176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:59.242715 (+ 5115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:59.242723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:59.242743 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:59.243544 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:59.244137 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:59.244140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:59.244142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:59.244466 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:59.245074 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:59.245082 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:59.245484 (+ 402us) client_negotiation.cc:769] Sending connection context
0926 20:52:59.245775 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 20:52:59.246357 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":358,"threads_started":1}
I20260926 20:52:59.267910 16868 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:42749
I20260926 20:52:59.267966 17003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:42749 every 8 connection(s)
I20260926 20:52:59.269392 16868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:52:59.271725 17004 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40349
I20260926 20:52:59.272094 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:59.256445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40349 (local address 127.8.249.129:39911)
0926 20:52:59.257294 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:59.257318 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:59.257492 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:59.257873 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:59.257876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:59.258333 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:59.258875 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:59.258895 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.260478 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.260485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:59.261520 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:59.261532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.262664 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.262672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:59.262870 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:59.263725 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:59.263758 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:59.265376 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:59.268888 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:59.268893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:59.268897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:59.269422 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:59.269808 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:59.269816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:59.269822 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:59.270020 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:59.270642 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:59.270650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:59.270954 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 20:52:59.271152 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 20:52:59.271535 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":442,"threads_started":1}
I20260926 20:52:59.272099 17004 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:59.272812 17004 heartbeater.cc:511] Master 127.8.249.190:40349 requested a full tablet report, sending...
I20260926 20:52:59.272930 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:59.255893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:39911 (local address 127.8.249.190:40349)
0926 20:52:59.256123 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:59.256131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:59.257433 (+ 1302us) server_negotiation.cc:407] Connection header received
0926 20:52:59.257621 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:59.257626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:59.257721 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:59.257843 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:59.259052 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.260295 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.261672 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.262437 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.262577 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:59.265598 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:59.265605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:59.265610 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:59.265652 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:59.268739 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:59.269552 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:59.269559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:59.269563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:59.269691 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:59.270143 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:59.270148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:59.270151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:59.270510 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:59.270669 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:59.272496 (+ 1827us) server_negotiation.cc:299] Negotiation successful
0926 20:52:59.272669 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:52:59.274863 16797 ts_manager.cc:194] Registered new tserver with Master: 2a90f39bc7eb4caea5d5d62edb3a52fd (127.8.249.129:42749)
I20260926 20:52:59.275507 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 16868
I20260926 20:52:59.275697 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:52:59.276492 9190 external_mini_cluster.cc:1428] Setting key 07ff4568a8a7e4fefb65f4f6dc2ab806
I20260926 20:52:59.276914 16797 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:48657
WARNING: no policy specified for oryx/127.8.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab.
Entry for principal oryx/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab.
I20260926 20:52:59.325835 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40349
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab
--principal=oryx/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:52:59.656426 17008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:52:59.656823 17008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:52:59.657004 17008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:52:59.664402 17008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:52:59.664556 17008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:52:59.664716 17008 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:52:59.664858 17008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:52:59.674772 17008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab
--principal=oryx/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.249.190:40349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:52:59.676136 17008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:52:59.677510 17008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:52:59.690399 17013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:59.690367 17014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:52:59.690949 17016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:52:59.691902 17008 server_base.cc:1034] running on GCE node
I20260926 20:52:59.692642 17008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:52:59.693650 17008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:52:59.694953 17008 hybrid_clock.cc:648] HybridClock initialized: now 1790455979694862 us; error 48 us; skew 500 ppm
I20260926 20:52:59.695304 17008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:52:59 dist-test-slave-2dg3 krb5kdc[16752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455979, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:52:59.698762 17008 init.cc:377] Logged in from keytab as oryx/127.8.249.130@KRBTEST.COM (short username oryx)
I20260926 20:52:59.700322 17008 webserver.cc:460] Webserver started at http://127.8.249.130:40203/ using document root <none> and password file <none>
I20260926 20:52:59.701113 17008 fs_manager.cc:359] Metadata directory not provided
I20260926 20:52:59.701256 17008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:52:59.701607 17008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:52:59.704851 17008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "32d34df039554a42acc3bd191ee23bb8"
format_stamp: "Formatted at 2026-09-26 20:52:59 on dist-test-slave-2dg3"
server_key: "e061729ec7e5394ce4314bb8dfcffcf1"
server_key_iv: "3a13ea89165b5cd2146b34c8700de2ff"
server_key_version: "encryptionkey@0"
I20260926 20:52:59.705749 17008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "32d34df039554a42acc3bd191ee23bb8"
format_stamp: "Formatted at 2026-09-26 20:52:59 on dist-test-slave-2dg3"
server_key: "e061729ec7e5394ce4314bb8dfcffcf1"
server_key_iv: "3a13ea89165b5cd2146b34c8700de2ff"
server_key_version: "encryptionkey@0"
I20260926 20:52:59.711544 17008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:52:59.715495 17023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:59.716915 17008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:52:59.717113 17008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "32d34df039554a42acc3bd191ee23bb8"
format_stamp: "Formatted at 2026-09-26 20:52:59 on dist-test-slave-2dg3"
server_key: "e061729ec7e5394ce4314bb8dfcffcf1"
server_key_iv: "3a13ea89165b5cd2146b34c8700de2ff"
server_key_version: "encryptionkey@0"
I20260926 20:52:59.717362 17008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:52:59.741617 17008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:52:59.746263 17008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:52:59.746556 17008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:52:59.749387 17008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:52:59.749514 17008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:52:59.749670 17008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:52:59.749749 17008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:52:59 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455979, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:52:59.773963 17032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:59.750736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40349 (local address 127.8.249.130:45313)
0926 20:52:59.751694 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:59.751743 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:59.753813 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:59.754611 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:59.754628 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:59.755513 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:59.756376 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:59.756419 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.758264 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.758269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:59.760326 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:59.760337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.761394 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.761401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:59.761605 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:59.762353 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:59.762374 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:59.765581 (+ 3207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:59.769257 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:59.769269 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:59.769285 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:59.769926 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:59.770402 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:59.770408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:59.770412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:59.770706 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:59.771520 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:59.771528 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:59.772060 (+ 532us) client_negotiation.cc:769] Sending connection context
0926 20:52:59.772425 (+ 365us) client_negotiation.cc:240] Negotiation successful
0926 20:52:59.772890 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":377,"threads_started":1}
I20260926 20:52:59.775082 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:59.750013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:45313 (local address 127.8.249.190:40349)
0926 20:52:59.750230 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:59.750236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:59.751888 (+ 1652us) server_negotiation.cc:407] Connection header received
0926 20:52:59.754114 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:59.754119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:59.754223 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:59.754337 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:59.756695 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.758030 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.760513 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.761147 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.761294 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:59.765809 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:59.765815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:59.765820 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:59.765863 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:59.769122 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:59.770126 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:59.770135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:59.770140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:59.770303 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:59.770892 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:59.770898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:59.770903 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:59.771285 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:59.771443 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:59.774530 (+ 3087us) server_negotiation.cc:299] Negotiation successful
0926 20:52:59.774738 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:52:59.797281 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:59.780938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40349 (local address 127.8.249.130:40149)
0926 20:52:59.781540 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 20:52:59.781556 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:52:59.781722 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:52:59.782058 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:52:59.782062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:52:59.782556 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:52:59.782935 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:59.782946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.784151 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.784160 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:59.786317 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:52:59.786339 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.787293 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.787308 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:52:59.787555 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:59.788368 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:52:59.788402 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:52:59.789828 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:52:59.794106 (+ 4278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:59.794112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:59.794115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:59.794780 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:59.795233 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:52:59.795238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:52:59.795241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:52:59.795471 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:52:59.796374 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:52:59.796379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:52:59.796570 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 20:52:59.796699 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 20:52:59.796949 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":352,"threads_started":1}
I20260926 20:52:59.797653 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:52:59.780844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:40149 (local address 127.8.249.190:40349)
0926 20:52:59.781011 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:52:59.781015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:52:59.781674 (+ 659us) server_negotiation.cc:407] Connection header received
0926 20:52:59.781860 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:52:59.781866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:52:59.781934 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:52:59.782036 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:52:59.783078 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.783991 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.786493 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:52:59.787079 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:52:59.787224 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:52:59.790077 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:52:59.790082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:52:59.790086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:52:59.790119 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:52:59.793948 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:59.794933 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:59.794938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:59.794941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:59.795090 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:52:59.795653 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:52:59.795663 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:52:59.795669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:52:59.796219 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:52:59.796410 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 20:52:59.796853 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 20:52:59.797162 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:52:59.804392 17008 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:45857
I20260926 20:52:59.804432 17142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:45857 every 8 connection(s)
I20260926 20:52:59.805584 17008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:52:59.807614 17143 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40349
I20260926 20:52:59.807845 17143 heartbeater.cc:464] Registering TS with master...
I20260926 20:52:59.808236 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 17008
I20260926 20:52:59.808410 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:52:59.808645 17143 heartbeater.cc:511] Master 127.8.249.190:40349 requested a full tablet report, sending...
I20260926 20:52:59.808990 9190 external_mini_cluster.cc:1428] Setting key ca4b58b4edcf1366ce1b6192f5e5d6db
I20260926 20:52:59.810194 16796 ts_manager.cc:194] Registered new tserver with Master: 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130:45857)
I20260926 20:52:59.811399 16796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:45313
WARNING: no policy specified for oryx/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab.
Entry for principal oryx/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab.
I20260926 20:52:59.858048 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40349
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab
--principal=oryx/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:00.189657 17147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:00.189953 17147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:00.190124 17147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:00.197388 17147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:00.197501 17147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:00.197688 17147 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:53:00.197808 17147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:53:00.207605 17147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab
--principal=oryx/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.249.190:40349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:00.208997 17147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:00.210386 17147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:00.223709 17152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:00.223768 17155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:00.224002 17153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:00.224741 17147 server_base.cc:1034] running on GCE node
I20260926 20:53:00.225436 17147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:00.226413 17147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:00.227648 17147 hybrid_clock.cc:648] HybridClock initialized: now 1790455980227572 us; error 47 us; skew 500 ppm
I20260926 20:53:00.227967 17147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:53:00 dist-test-slave-2dg3 krb5kdc[16752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455980, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:00.231148 17147 init.cc:377] Logged in from keytab as oryx/127.8.249.131@KRBTEST.COM (short username oryx)
I20260926 20:53:00.232779 17147 webserver.cc:460] Webserver started at http://127.8.249.131:41813/ using document root <none> and password file <none>
I20260926 20:53:00.233475 17147 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:00.233587 17147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:00.233894 17147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:00.236814 17147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "adaf6826ac464b79af323870bd2ca188"
format_stamp: "Formatted at 2026-09-26 20:53:00 on dist-test-slave-2dg3"
server_key: "cec0e181099a50b1d94be4a48ddf25b2"
server_key_iv: "49174b211e345f87a44f6b82349a51ba"
server_key_version: "encryptionkey@0"
I20260926 20:53:00.237627 17147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "adaf6826ac464b79af323870bd2ca188"
format_stamp: "Formatted at 2026-09-26 20:53:00 on dist-test-slave-2dg3"
server_key: "cec0e181099a50b1d94be4a48ddf25b2"
server_key_iv: "49174b211e345f87a44f6b82349a51ba"
server_key_version: "encryptionkey@0"
I20260926 20:53:00.242733 17147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 20:53:00.246224 17162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:00.247817 17147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:53:00.248013 17147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "adaf6826ac464b79af323870bd2ca188"
format_stamp: "Formatted at 2026-09-26 20:53:00 on dist-test-slave-2dg3"
server_key: "cec0e181099a50b1d94be4a48ddf25b2"
server_key_iv: "49174b211e345f87a44f6b82349a51ba"
server_key_version: "encryptionkey@0"
I20260926 20:53:00.248262 17147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:00.271307 17147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:00.276130 17147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:00.276402 17147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:00.279166 17147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:00.279273 17147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:00.279403 17147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:00.279484 17147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:00.279927 17004 heartbeater.cc:503] Master 127.8.249.190:40349 was elected leader, sending a full tablet report...
Sep 26 20:53:00 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455980, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:53:00.301956 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:00.279893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:34833 (local address 127.8.249.190:40349)
0926 20:53:00.280078 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:00.280084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:00.281761 (+ 1677us) server_negotiation.cc:407] Connection header received
0926 20:53:00.283838 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:00.283843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:00.283933 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:00.284037 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:00.286189 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:00.287330 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:00.289032 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:00.289424 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:00.289553 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:00.294214 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:00.294219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:00.294223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:00.294253 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:00.297961 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:00.299070 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:00.299074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:00.299076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:00.299208 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:00.299761 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:00.299765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:00.299767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:00.300009 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:00.300114 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:00.301239 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0926 20:53:00.301508 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 20:53:00.304239 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:00.280627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40349 (local address 127.8.249.131:34833)
0926 20:53:00.281600 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:00.281651 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:00.283564 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:00.284284 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:00.284291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:00.285030 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:00.286015 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:00.286035 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:00.287467 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:00.287472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:00.288887 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:00.288899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:00.289656 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:00.289663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:00.289863 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:00.290516 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:00.290540 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:00.293963 (+ 3423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:00.298115 (+ 4152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:00.298130 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:00.298153 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:00.298930 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:00.299326 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:00.299333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:00.299339 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:00.299642 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:00.300134 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:00.300146 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:00.300701 (+ 555us) client_negotiation.cc:769] Sending connection context
0926 20:53:00.301081 (+ 380us) client_negotiation.cc:240] Negotiation successful
0926 20:53:00.301773 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":381,"threads_started":1}
I20260926 20:53:00.340642 17147 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:36937
I20260926 20:53:00.340656 17281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:36937 every 8 connection(s)
I20260926 20:53:00.341686 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:00.325181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40349 (local address 127.8.249.131:42199)
0926 20:53:00.325875 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:00.325896 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:00.326053 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:00.326500 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:00.326503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:00.327187 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:00.327689 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:00.327721 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:00.328989 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:00.328994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:00.330931 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:00.330951 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:00.332316 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:00.332321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:00.332445 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:00.333223 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:00.333239 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:00.333938 (+ 699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:00.338509 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:00.338519 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:00.338523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:00.339032 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:00.339519 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:00.339524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:00.339527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:00.339721 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:00.340497 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:00.340522 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:00.340747 (+ 225us) client_negotiation.cc:769] Sending connection context
0926 20:53:00.340894 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 20:53:00.341232 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":375,"threads_started":1}
I20260926 20:53:00.342325 17147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:53:00.342674 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:00.325044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:42199 (local address 127.8.249.190:40349)
0926 20:53:00.325281 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:00.325290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:00.326000 (+ 710us) server_negotiation.cc:407] Connection header received
0926 20:53:00.326198 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:00.326206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:00.326319 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:00.326495 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:00.327947 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:00.328842 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:00.331137 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:00.332107 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:00.332297 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:00.334184 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:00.334195 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:00.334203 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:00.334257 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:00.338350 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:00.339155 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:00.339163 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:00.339167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:00.339332 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:00.339899 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:00.339905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:00.339911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:00.340322 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:00.341955 (+ 1633us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:00.342083 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 20:53:00.342316 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 20:53:00.344470 17282 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40349
I20260926 20:53:00.344718 17282 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:00.345510 17282 heartbeater.cc:511] Master 127.8.249.190:40349 requested a full tablet report, sending...
I20260926 20:53:00.346946 16796 ts_manager.cc:194] Registered new tserver with Master: adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937)
I20260926 20:53:00.347741 16796 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:34833
I20260926 20:53:00.350118 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 17147
I20260926 20:53:00.350255 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:53:00.350637 9190 external_mini_cluster.cc:1428] Setting key e4eacbab23b07a9bf361ce8ea7f50f98
I20260926 20:53:00.353624 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:53:00 dist-test-slave-2dg3 krb5kdc[16752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455980, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:53:00 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:00 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:00.813959 17143 heartbeater.cc:503] Master 127.8.249.190:40349 was elected leader, sending a full tablet report...
I20260926 20:53:01.350225 17282 heartbeater.cc:503] Master 127.8.249.190:40349 was elected leader, sending a full tablet report...
W20260926 20:53:01.455170 17290 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.055s user 0.007s sys 0.007s
W20260926 20:53:01.456458 16892 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:00.394289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.8.249.190:40349)
0926 20:53:00.394468 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:00.394473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:00.394754 (+ 281us) server_negotiation.cc:407] Connection header received
0926 20:53:00.394988 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:00.394993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:00.395058 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:00.395155 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:00.396447 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:00.397148 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:00.398316 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:00.398720 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:00.398866 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:01.456202 (+1057336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36588: BlockingRecv error: recv got EOF from 127.0.0.1:36588 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 20:53:01 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455980, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:53:01.482633 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:01.464191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36596 (local address 127.8.249.190:40349)
0926 20:53:01.464442 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:01.464449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:01.464730 (+ 281us) server_negotiation.cc:407] Connection header received
0926 20:53:01.464879 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:01.464884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:01.464977 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:01.465069 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:01.466295 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.467083 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.468398 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.468810 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.468900 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:01.477521 (+ 8621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:01.477526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:01.477530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:01.477565 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:01.480216 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:01.480882 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:01.480886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:01.480888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:01.481000 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:01.481369 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:01.481372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:01.481374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:01.481629 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:01.481841 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:01.482122 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 20:53:01.482308 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:53:01.487005 16796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:53:01.493841 16796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:53:01.518774 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:01.507006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.8.249.131:36937)
0926 20:53:01.508328 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:01.508340 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:01.508376 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:53:01.508499 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:01.508511 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:01.508856 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:01.509979 (+ 1123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:01.510248 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.512204 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.516125 (+ 3921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.517566 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.517738 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:01.517971 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:01.518286 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 20:53:01.518462 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":1285,"threads_started":1}
I20260926 20:53:01.518775 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:01.506439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:36937 (local address 127.0.0.1:46698)
0926 20:53:01.507335 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:01.507394 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:01.507621 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:01.509155 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:01.509166 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:01.509210 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:01.509722 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:01.509741 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.512851 (+ 3110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.512858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:01.515856 (+ 2998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:01.515880 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.517785 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.517793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:01.518018 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:01.518071 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 20:53:01.518215 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 20:53:01.518406 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":459,"threads_started":1}
I20260926 20:53:01.522964 17213 tablet_service.cc:1467] Processing CreateTablet for tablet 2f51d9396a4043afbccc5aaaae8ba980 (DEFAULT_TABLE table=test-table [id=8e9f19d127cc4877ae5f9eac2dd7b6ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:01.524652 17213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f51d9396a4043afbccc5aaaae8ba980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:01.525113 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:01.506607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:42749 (local address 127.0.0.1:38310)
0926 20:53:01.515148 (+ 8541us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:01.515164 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:01.515325 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:01.516834 (+ 1509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:01.516838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:01.516862 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:01.517289 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:01.517304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.519344 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.519349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:01.522048 (+ 2699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:01.522061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.524123 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.524132 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:01.524333 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:01.524368 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:53:01.524534 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 20:53:01.524708 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8428,"mutex_wait_us":1,"thread_start_us":9476,"threads_started":1}
I20260926 20:53:01.525252 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:01.507738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:45857 (local address 127.0.0.1:38836)
0926 20:53:01.516361 (+ 8623us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:01.516378 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:01.516492 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:01.517219 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:01.517224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:01.517257 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:01.517612 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:01.517624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.519422 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.519429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:01.522224 (+ 2795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:01.522239 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.524414 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.524421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:01.524600 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:01.524631 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 20:53:01.524724 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 20:53:01.524845 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8508,"mutex_wait_us":341,"thread_start_us":5066,"threads_started":1}
I20260926 20:53:01.525677 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:01.507047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.8.249.129:42749)
0926 20:53:01.512724 (+ 5677us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:01.512737 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:01.515291 (+ 2554us) server_negotiation.cc:407] Connection header received
0926 20:53:01.515563 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:01.515571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:01.516440 (+ 869us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:01.516663 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:01.517467 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.519202 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.522230 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.523851 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.523999 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:01.524216 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:01.524584 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 20:53:01.524780 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5499,"thread_start_us":5381,"threads_started":1}
I20260926 20:53:01.525941 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:01.508064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38836 (local address 127.8.249.130:45857)
0926 20:53:01.515920 (+ 7856us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:01.515930 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:01.516620 (+ 690us) server_negotiation.cc:407] Connection header received
0926 20:53:01.516733 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:01.516740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:01.516997 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:01.517222 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:01.517806 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.519292 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.522397 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.523851 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.523999 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:01.524205 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:01.525452 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0926 20:53:01.525636 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7499,"thread_start_us":7341,"threads_started":1}
I20260926 20:53:01.529031 16934 tablet_service.cc:1467] Processing CreateTablet for tablet 2f51d9396a4043afbccc5aaaae8ba980 (DEFAULT_TABLE table=test-table [id=8e9f19d127cc4877ae5f9eac2dd7b6ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:01.529460 17073 tablet_service.cc:1467] Processing CreateTablet for tablet 2f51d9396a4043afbccc5aaaae8ba980 (DEFAULT_TABLE table=test-table [id=8e9f19d127cc4877ae5f9eac2dd7b6ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:01.530349 16934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f51d9396a4043afbccc5aaaae8ba980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:01.530871 17073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f51d9396a4043afbccc5aaaae8ba980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:01.541625 17307 tablet_bootstrap.cc:492] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188: Bootstrap starting.
I20260926 20:53:01.543671 17308 tablet_bootstrap.cc:492] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Bootstrap starting.
I20260926 20:53:01.545181 17309 tablet_bootstrap.cc:492] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8: Bootstrap starting.
I20260926 20:53:01.546060 17307 tablet_bootstrap.cc:654] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:01.547326 17308 tablet_bootstrap.cc:654] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:01.547457 17307 log.cc:824] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:01.548628 17308 log.cc:824] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:01.548897 17309 tablet_bootstrap.cc:654] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:01.550468 17309 log.cc:824] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:01.554522 17309 tablet_bootstrap.cc:492] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8: No bootstrap required, opened a new log
I20260926 20:53:01.554764 17307 tablet_bootstrap.cc:492] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188: No bootstrap required, opened a new log
I20260926 20:53:01.554862 17309 ts_tablet_manager.cc:1397] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 20:53:01.554916 17308 tablet_bootstrap.cc:492] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd: No bootstrap required, opened a new log
I20260926 20:53:01.555094 17307 ts_tablet_manager.cc:1397] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.002s
I20260926 20:53:01.555182 17308 ts_tablet_manager.cc:1397] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260926 20:53:01.563740 17307 raft_consensus.cc:348] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } }
I20260926 20:53:01.564184 17307 raft_consensus.cc:374] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:01.564328 17307 raft_consensus.cc:729] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adaf6826ac464b79af323870bd2ca188, State: Initialized, Role: FOLLOWER
I20260926 20:53:01.564997 17307 consensus_queue.cc:260] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } }
I20260926 20:53:01.565960 17309 raft_consensus.cc:348] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } }
I20260926 20:53:01.565960 17308 raft_consensus.cc:348] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } }
I20260926 20:53:01.566419 17308 raft_consensus.cc:374] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:01.566485 17309 raft_consensus.cc:374] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:01.566571 17308 raft_consensus.cc:729] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a90f39bc7eb4caea5d5d62edb3a52fd, State: Initialized, Role: FOLLOWER
I20260926 20:53:01.566630 17309 raft_consensus.cc:729] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d34df039554a42acc3bd191ee23bb8, State: Initialized, Role: FOLLOWER
I20260926 20:53:01.566895 17307 ts_tablet_manager.cc:1428] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260926 20:53:01.567373 17309 consensus_queue.cc:260] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } }
I20260926 20:53:01.567373 17308 consensus_queue.cc:260] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } }
I20260926 20:53:01.569519 17309 ts_tablet_manager.cc:1428] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 20:53:01.572479 17308 ts_tablet_manager.cc:1428] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
W20260926 20:53:01.595039 17283 tablet.cc:2369] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:53:01.723944 17313 raft_consensus.cc:482] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:01.724357 17313 raft_consensus.cc:504] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } }
I20260926 20:53:01.726217 17313 leader_election.cc:288] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a90f39bc7eb4caea5d5d62edb3a52fd (127.8.249.129:42749), 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130:45857)
I20260926 20:53:01.733346 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:01.726547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:38499 (local address 127.8.249.129:42749)
0926 20:53:01.726825 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:01.726835 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:01.727627 (+ 792us) server_negotiation.cc:407] Connection header received
0926 20:53:01.727868 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:01.727872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:01.727931 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:01.728028 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:01.728705 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.729418 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.731554 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.732353 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.732547 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:01.732636 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:01.732938 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 20:53:01.733067 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:53:01.733389 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:01.726894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:42063 (local address 127.8.249.130:45857)
0926 20:53:01.727113 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:01.727121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:01.727601 (+ 480us) server_negotiation.cc:407] Connection header received
0926 20:53:01.727857 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:01.727863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:01.727930 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:01.728028 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:01.728670 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.729410 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.731623 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.732419 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.732625 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:01.732703 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:01.732995 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 20:53:01.733127 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:53:01.733601 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:01.726490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:45857 (local address 127.8.249.131:42063)
0926 20:53:01.727487 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:01.727506 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:01.727703 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:01.728042 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:01.728046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:01.728091 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:01.728530 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:01.728539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.729538 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.729546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:01.731401 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:01.731414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.732659 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.732665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:01.732839 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:01.732862 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:53:01.732960 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:53:01.733090 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":21,"spinlock_wait_cycles":29184,"thread_start_us":593,"threads_started":1}
I20260926 20:53:01.733623 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:01.726568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:42749 (local address 127.8.249.131:38499)
0926 20:53:01.727515 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:01.727530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:01.727712 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:01.728043 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:01.728047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:01.728094 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:01.728546 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:01.728556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.729556 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.729562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:01.731367 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:01.731385 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.732612 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.732619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:01.732768 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:01.732796 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 20:53:01.732910 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 20:53:01.733055 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":52,"spinlock_wait_cycles":41216,"thread_start_us":596,"threads_started":1}
I20260926 20:53:01.734912 16954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f51d9396a4043afbccc5aaaae8ba980" candidate_uuid: "adaf6826ac464b79af323870bd2ca188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" is_pre_election: true
I20260926 20:53:01.734912 17098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f51d9396a4043afbccc5aaaae8ba980" candidate_uuid: "adaf6826ac464b79af323870bd2ca188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d34df039554a42acc3bd191ee23bb8" is_pre_election: true
I20260926 20:53:01.735526 16954 raft_consensus.cc:2454] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adaf6826ac464b79af323870bd2ca188 in term 0.
I20260926 20:53:01.735526 17098 raft_consensus.cc:2454] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adaf6826ac464b79af323870bd2ca188 in term 0.
I20260926 20:53:01.736282 17163 leader_election.cc:302] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a90f39bc7eb4caea5d5d62edb3a52fd, 32d34df039554a42acc3bd191ee23bb8, adaf6826ac464b79af323870bd2ca188; no voters:
I20260926 20:53:01.736776 17313 raft_consensus.cc:2781] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:53:01.736930 17313 raft_consensus.cc:482] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:53:01.737025 17313 raft_consensus.cc:3037] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:01.739323 17313 raft_consensus.cc:504] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } }
I20260926 20:53:01.739950 17313 leader_election.cc:288] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [CANDIDATE]: Term 1 election: Requested vote from peers 2a90f39bc7eb4caea5d5d62edb3a52fd (127.8.249.129:42749), 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130:45857)
I20260926 20:53:01.740568 17098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f51d9396a4043afbccc5aaaae8ba980" candidate_uuid: "adaf6826ac464b79af323870bd2ca188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d34df039554a42acc3bd191ee23bb8"
I20260926 20:53:01.740571 16954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f51d9396a4043afbccc5aaaae8ba980" candidate_uuid: "adaf6826ac464b79af323870bd2ca188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd"
I20260926 20:53:01.740787 17098 raft_consensus.cc:3037] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:01.740787 16954 raft_consensus.cc:3037] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:01.744108 16954 raft_consensus.cc:2454] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adaf6826ac464b79af323870bd2ca188 in term 1.
I20260926 20:53:01.744108 17098 raft_consensus.cc:2454] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adaf6826ac464b79af323870bd2ca188 in term 1.
I20260926 20:53:01.744704 17166 leader_election.cc:302] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d34df039554a42acc3bd191ee23bb8, adaf6826ac464b79af323870bd2ca188; no voters:
I20260926 20:53:01.745126 17313 raft_consensus.cc:2781] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:01.745589 17313 raft_consensus.cc:686] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [term 1 LEADER]: Becoming Leader. State: Replica: adaf6826ac464b79af323870bd2ca188, State: Running, Role: LEADER
I20260926 20:53:01.746158 17313 consensus_queue.cc:237] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } }
I20260926 20:53:01.750962 16795 catalog_manager.cc:5495] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 reported cstate change: term changed from 0 to 1, leader changed from <none> to adaf6826ac464b79af323870bd2ca188 (127.8.249.131). New cstate: current_term: 1 leader_uuid: "adaf6826ac464b79af323870bd2ca188" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } health_report { overall_health: UNKNOWN } } }
W20260926 20:53:01.773813 17005 tablet.cc:2369] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:53:01.808437 17144 tablet.cc:2369] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:53:01.818362 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:01.812286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.8.249.131:36937)
0926 20:53:01.812512 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:01.812518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:01.812538 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:53:01.812644 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:01.812650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:01.812760 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:01.812917 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:01.813247 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.814341 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.816310 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:01.816716 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:01.816818 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:01.817212 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:01.817786 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:01.817892 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:01.818024 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 20:53:01.818094 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:53:01.827231 16954 raft_consensus.cc:1261] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Refusing update from remote peer adaf6826ac464b79af323870bd2ca188: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:01.828233 17313 consensus_queue.cc:1035] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:01.828438 17098 raft_consensus.cc:1261] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Refusing update from remote peer adaf6826ac464b79af323870bd2ca188: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:01.829214 17313 consensus_queue.cc:1035] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:01.846107 17321 mvcc.cc:205] Tried to move back new op lower bound from 7333707701553242112 to 7333707701236846592. Current Snapshot: MvccSnapshot[applied={T|T < 7333707701553242112}]
I20260926 20:53:01.861534 16795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:01.861858 16795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:01.866065 16795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585: Sending DeleteTablet for 3 replicas of tablet 2f51d9396a4043afbccc5aaaae8ba980
I20260926 20:53:01.867497 17213 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f51d9396a4043afbccc5aaaae8ba980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:01 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:46698
I20260926 20:53:01.867962 16934 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f51d9396a4043afbccc5aaaae8ba980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:01 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:38310
I20260926 20:53:01.868681 17331 tablet_replica.cc:330] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188: stopping tablet replica
I20260926 20:53:01.868670 17073 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f51d9396a4043afbccc5aaaae8ba980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:01 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:38836
I20260926 20:53:01.869169 17331 raft_consensus.cc:2229] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:53:01.870119 17331 raft_consensus.cc:2258] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:01.870177 16795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:53:01.870937 16795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:53:01.877725 17331 ts_tablet_manager.cc:1905] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:01.880280 17213 tablet_service.cc:1467] Processing CreateTablet for tablet 8283f6a0765e431eb29a52582c0731f7 (DEFAULT_TABLE table=test-table [id=d4d39100be89420dac5cf208519c675f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:01.880791 17213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8283f6a0765e431eb29a52582c0731f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:01.880792 17072 tablet_service.cc:1467] Processing CreateTablet for tablet 8283f6a0765e431eb29a52582c0731f7 (DEFAULT_TABLE table=test-table [id=d4d39100be89420dac5cf208519c675f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:01.881328 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8283f6a0765e431eb29a52582c0731f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:01.883948 16933 tablet_service.cc:1467] Processing CreateTablet for tablet 8283f6a0765e431eb29a52582c0731f7 (DEFAULT_TABLE table=test-table [id=d4d39100be89420dac5cf208519c675f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:01.884714 16933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8283f6a0765e431eb29a52582c0731f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:01.888016 17309 tablet_bootstrap.cc:492] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8: Bootstrap starting.
I20260926 20:53:01.888581 17332 tablet_replica.cc:330] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd: stopping tablet replica
I20260926 20:53:01.889052 17332 raft_consensus.cc:2229] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:01.889601 17332 raft_consensus.cc:2258] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:01.889750 17309 tablet_bootstrap.cc:654] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:01.892614 17308 tablet_bootstrap.cc:492] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Bootstrap starting.
I20260926 20:53:01.893061 17307 tablet_bootstrap.cc:492] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188: Bootstrap starting.
I20260926 20:53:01.894358 17308 tablet_bootstrap.cc:654] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:01.895258 17307 tablet_bootstrap.cc:654] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:01.905160 17308 tablet_bootstrap.cc:492] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd: No bootstrap required, opened a new log
I20260926 20:53:01.905548 17308 ts_tablet_manager.cc:1397] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20260926 20:53:01.906091 17331 ts_tablet_manager.cc:1918] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:01.906536 17308 raft_consensus.cc:348] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:01.906903 17308 raft_consensus.cc:374] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:01.907047 17308 raft_consensus.cc:729] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a90f39bc7eb4caea5d5d62edb3a52fd, State: Initialized, Role: FOLLOWER
I20260926 20:53:01.907352 17309 tablet_bootstrap.cc:492] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8: No bootstrap required, opened a new log
I20260926 20:53:01.907548 17309 ts_tablet_manager.cc:1397] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20260926 20:53:01.907485 17308 consensus_queue.cc:260] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:01.908802 17309 raft_consensus.cc:348] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:01.909107 17309 raft_consensus.cc:374] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:01.909250 17309 raft_consensus.cc:729] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d34df039554a42acc3bd191ee23bb8, State: Initialized, Role: FOLLOWER
I20260926 20:53:01.909677 17309 consensus_queue.cc:260] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:01.910526 17332 ts_tablet_manager.cc:1905] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:01.911012 17308 ts_tablet_manager.cc:1428] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 20:53:01.913666 17309 ts_tablet_manager.cc:1428] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260926 20:53:01.913822 17331 log.cc:1196] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/2f51d9396a4043afbccc5aaaae8ba980
I20260926 20:53:01.914495 17331 ts_tablet_manager.cc:1939] T 2f51d9396a4043afbccc5aaaae8ba980 P adaf6826ac464b79af323870bd2ca188: Deleting consensus metadata
I20260926 20:53:01.916467 16784 catalog_manager.cc:4841] TS adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937): tablet 2f51d9396a4043afbccc5aaaae8ba980 (table test-table [id=8e9f19d127cc4877ae5f9eac2dd7b6ed]) successfully deleted
I20260926 20:53:01.917912 17333 tablet_replica.cc:330] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8: stopping tablet replica
I20260926 20:53:01.918202 17307 tablet_bootstrap.cc:492] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188: No bootstrap required, opened a new log
I20260926 20:53:01.918362 17333 raft_consensus.cc:2229] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:01.918407 17307 ts_tablet_manager.cc:1397] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.004s
I20260926 20:53:01.918896 17333 raft_consensus.cc:2258] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:01.919489 17307 raft_consensus.cc:348] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:01.919843 17307 raft_consensus.cc:374] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:01.920027 17307 raft_consensus.cc:729] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adaf6826ac464b79af323870bd2ca188, State: Initialized, Role: FOLLOWER
I20260926 20:53:01.920473 17307 consensus_queue.cc:260] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:01.921128 17333 ts_tablet_manager.cc:1905] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:01.923710 17307 ts_tablet_manager.cc:1428] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260926 20:53:01.924646 17332 ts_tablet_manager.cc:1918] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:01.924870 17332 log.cc:1196] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/2f51d9396a4043afbccc5aaaae8ba980
I20260926 20:53:01.925803 17332 ts_tablet_manager.cc:1939] T 2f51d9396a4043afbccc5aaaae8ba980 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Deleting consensus metadata
I20260926 20:53:01.927870 16781 catalog_manager.cc:4841] TS 2a90f39bc7eb4caea5d5d62edb3a52fd (127.8.249.129:42749): tablet 2f51d9396a4043afbccc5aaaae8ba980 (table test-table [id=8e9f19d127cc4877ae5f9eac2dd7b6ed]) successfully deleted
I20260926 20:53:01.929050 17333 ts_tablet_manager.cc:1918] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:01.929234 17333 log.cc:1196] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/2f51d9396a4043afbccc5aaaae8ba980
I20260926 20:53:01.929754 17333 ts_tablet_manager.cc:1939] T 2f51d9396a4043afbccc5aaaae8ba980 P 32d34df039554a42acc3bd191ee23bb8: Deleting consensus metadata
I20260926 20:53:01.931500 16784 catalog_manager.cc:4841] TS 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130:45857): tablet 2f51d9396a4043afbccc5aaaae8ba980 (table test-table [id=8e9f19d127cc4877ae5f9eac2dd7b6ed]) successfully deleted
I20260926 20:53:02.262368 17314 raft_consensus.cc:482] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:02.262620 17314 raft_consensus.cc:504] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.264263 17314 leader_election.cc:288] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130:45857), adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937)
I20260926 20:53:02.273684 17315 raft_consensus.cc:482] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:02.273927 17315 raft_consensus.cc:504] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.275650 17315 leader_election.cc:288] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a90f39bc7eb4caea5d5d62edb3a52fd (127.8.249.129:42749), adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937)
I20260926 20:53:02.295915 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.276492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:33035 (local address 127.8.249.131:36937)
0926 20:53:02.285860 (+ 9368us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:02.285869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:02.289147 (+ 3278us) server_negotiation.cc:407] Connection header received
0926 20:53:02.289338 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:02.289342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:02.289410 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:02.289502 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:02.290030 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.291185 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.293602 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.294686 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.294836 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.294927 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:02.295418 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 20:53:02.295566 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9205,"thread_start_us":9047,"threads_started":1}
I20260926 20:53:02.296643 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.276119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:36937 (local address 127.8.249.130:33035)
0926 20:53:02.289033 (+ 12914us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:02.289057 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:02.289225 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:02.289553 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:02.289558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:02.289584 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:02.289835 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.289845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.291382 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.291389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.293401 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.293417 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.295061 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.295067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.295235 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.295264 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 20:53:02.296168 (+ 904us) client_negotiation.cc:240] Negotiation successful
0926 20:53:02.296315 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12774,"thread_start_us":13195,"threads_started":1}
I20260926 20:53:02.297583 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8283f6a0765e431eb29a52582c0731f7" candidate_uuid: "32d34df039554a42acc3bd191ee23bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adaf6826ac464b79af323870bd2ca188" is_pre_election: true
I20260926 20:53:02.298141 17234 raft_consensus.cc:2454] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32d34df039554a42acc3bd191ee23bb8 in term 0.
I20260926 20:53:02.298964 17027 leader_election.cc:302] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d34df039554a42acc3bd191ee23bb8, adaf6826ac464b79af323870bd2ca188; no voters:
I20260926 20:53:02.299443 17315 raft_consensus.cc:2781] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:53:02.299585 17315 raft_consensus.cc:482] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:53:02.299700 17315 raft_consensus.cc:3037] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:02.302342 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.265641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:51579 (local address 127.8.249.131:36937)
0926 20:53:02.265881 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:02.265890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:02.286280 (+ 20390us) server_negotiation.cc:407] Connection header received
0926 20:53:02.286557 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:02.286563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:02.286655 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:02.286717 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:02.287710 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.288860 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.297951 (+ 9091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.299212 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.299370 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.299446 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:02.301895 (+ 2449us) server_negotiation.cc:299] Negotiation successful
0926 20:53:02.302040 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 20:53:02.302405 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.265251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:36937 (local address 127.8.249.129:51579)
0926 20:53:02.286143 (+ 20892us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:02.286164 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:02.286399 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:02.286940 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:02.286944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:02.286974 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:02.287326 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.287338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.289067 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.289072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.291264 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.291279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.301469 (+ 10190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.301477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.301705 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.301748 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:53:02.301840 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:53:02.302001 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20767,"mutex_wait_us":1,"thread_start_us":20624,"threads_started":1}
I20260926 20:53:02.303071 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8283f6a0765e431eb29a52582c0731f7" candidate_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adaf6826ac464b79af323870bd2ca188" is_pre_election: true
I20260926 20:53:02.303328 17234 raft_consensus.cc:2454] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a90f39bc7eb4caea5d5d62edb3a52fd in term 0.
I20260926 20:53:02.303421 17315 raft_consensus.cc:504] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.303882 16887 leader_election.cc:302] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a90f39bc7eb4caea5d5d62edb3a52fd, adaf6826ac464b79af323870bd2ca188; no voters:
I20260926 20:53:02.304389 17314 raft_consensus.cc:2781] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:53:02.304687 17314 raft_consensus.cc:482] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:53:02.304772 17314 raft_consensus.cc:3037] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:02.305155 17315 leader_election.cc:288] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [CANDIDATE]: Term 1 election: Requested vote from peers 2a90f39bc7eb4caea5d5d62edb3a52fd (127.8.249.129:42749), adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937)
I20260926 20:53:02.305256 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.264974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:58765 (local address 127.8.249.130:45857)
0926 20:53:02.293821 (+ 28847us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:02.293827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:02.293845 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:53:02.293900 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:02.293904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:02.293969 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:02.294069 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:02.298035 (+ 3966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.299286 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.301698 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.303291 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.304650 (+ 1359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.304720 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:02.304769 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 20:53:02.304902 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28716,"thread_start_us":29279,"threads_started":1}
I20260926 20:53:02.305588 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8283f6a0765e431eb29a52582c0731f7" candidate_uuid: "32d34df039554a42acc3bd191ee23bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adaf6826ac464b79af323870bd2ca188"
I20260926 20:53:02.305794 17234 raft_consensus.cc:3037] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:02.307657 17314 raft_consensus.cc:504] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.308395 17314 leader_election.cc:288] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [CANDIDATE]: Term 1 election: Requested vote from peers 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130:45857), adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937)
I20260926 20:53:02.308897 17233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8283f6a0765e431eb29a52582c0731f7" candidate_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adaf6826ac464b79af323870bd2ca188"
I20260926 20:53:02.309124 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.264565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:45857 (local address 127.8.249.129:58765)
0926 20:53:02.265090 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:02.265114 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:02.265301 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:02.294091 (+ 28790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:02.294095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:02.294131 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:02.294502 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.294519 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.299448 (+ 4929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.299453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.301488 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.301512 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.303555 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.303565 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.303761 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.303850 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 20:53:02.303946 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 20:53:02.304108 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":24,"thread_start_us":241,"threads_started":1}
I20260926 20:53:02.308974 17234 raft_consensus.cc:2454] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32d34df039554a42acc3bd191ee23bb8 in term 1.
I20260926 20:53:02.309969 17098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8283f6a0765e431eb29a52582c0731f7" candidate_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d34df039554a42acc3bd191ee23bb8" is_pre_election: true
I20260926 20:53:02.310175 17027 leader_election.cc:302] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d34df039554a42acc3bd191ee23bb8, adaf6826ac464b79af323870bd2ca188; no voters:
I20260926 20:53:02.310303 17098 raft_consensus.cc:2379] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2a90f39bc7eb4caea5d5d62edb3a52fd in current term 1: Already voted for candidate 32d34df039554a42acc3bd191ee23bb8 in this term.
I20260926 20:53:02.312042 17097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8283f6a0765e431eb29a52582c0731f7" candidate_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d34df039554a42acc3bd191ee23bb8"
I20260926 20:53:02.312393 17315 raft_consensus.cc:2781] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:02.312870 17315 raft_consensus.cc:686] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 1 LEADER]: Becoming Leader. State: Replica: 32d34df039554a42acc3bd191ee23bb8, State: Running, Role: LEADER
I20260926 20:53:02.313289 17315 consensus_queue.cc:237] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.314420 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.294863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:35599 (local address 127.8.249.129:42749)
0926 20:53:02.303007 (+ 8144us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:02.303013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:02.303035 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:53:02.303093 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:02.303098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:02.303186 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:02.303758 (+ 572us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:02.304107 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.310158 (+ 6051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.312420 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.313381 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.313537 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.313624 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:02.313967 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 20:53:02.314082 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":7982,"thread_start_us":9647,"threads_started":1}
I20260926 20:53:02.314435 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.297387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:42749 (local address 127.8.249.130:35599)
0926 20:53:02.297645 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:02.297663 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:02.297765 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:02.303346 (+ 5581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:02.303354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:02.303387 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:02.303624 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.303636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.310309 (+ 6673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.310314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.311809 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.311825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.313575 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.313583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.313795 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.313829 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:53:02.313935 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 20:53:02.314097 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 20:53:02.315261 16954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8283f6a0765e431eb29a52582c0731f7" candidate_uuid: "32d34df039554a42acc3bd191ee23bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" is_pre_election: true
I20260926 20:53:02.315436 16953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8283f6a0765e431eb29a52582c0731f7" candidate_uuid: "32d34df039554a42acc3bd191ee23bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd"
I20260926 20:53:02.315613 16954 raft_consensus.cc:2379] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32d34df039554a42acc3bd191ee23bb8 in current term 1: Already voted for candidate 2a90f39bc7eb4caea5d5d62edb3a52fd in this term.
I20260926 20:53:02.316593 16887 leader_election.cc:302] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a90f39bc7eb4caea5d5d62edb3a52fd; no voters: 32d34df039554a42acc3bd191ee23bb8, adaf6826ac464b79af323870bd2ca188
I20260926 20:53:02.316917 17314 raft_consensus.cc:2726] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 20:53:02.317621 16797 catalog_manager.cc:5495] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130). New cstate: current_term: 1 leader_uuid: "32d34df039554a42acc3bd191ee23bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:53:02 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455978, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.190@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:53:02.383296 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.369289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.8.249.190:40349)
0926 20:53:02.369897 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:02.369901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:02.370317 (+ 416us) server_negotiation.cc:407] Connection header received
0926 20:53:02.370461 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:02.370464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:02.370541 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:02.370643 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:02.371772 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.372538 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.373739 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.374144 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.374233 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.377729 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:02.377735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:02.377738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:02.377768 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:02.380881 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:02.381633 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:02.381637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:02.381639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:02.381760 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:02.382085 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:02.382088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:02.382090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:02.382362 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:02.382485 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:02.382859 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 20:53:02.383030 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":344,"threads_started":1}
I20260926 20:53:02.383585 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.369242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40349 (local address 127.0.0.1:36606)
0926 20:53:02.370206 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:02.370225 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:02.370359 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:02.370730 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:02.370733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:02.371446 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:02.371632 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.371640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.372668 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.372672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.373618 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.373627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.374352 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.374356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.374469 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.375127 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:02.375143 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:02.377512 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:02.381050 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:02.381064 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:02.381070 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:02.381516 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:02.381856 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:02.381859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:02.381861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:02.381995 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:02.382466 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:02.382479 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:02.382620 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 20:53:02.382758 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 20:53:02.383161 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":89,"thread_start_us":749,"threads_started":1}
I20260926 20:53:02.388978 16797 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:36606:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:53:02.390012 16797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:53:02.397641 17213 tablet_service.cc:1467] Processing CreateTablet for tablet de8be0f60d434e278e7e8d8ae37e05b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55d3d8857b6c4c93a71fe1c1b65d0327]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:53:02.398322 17213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de8be0f60d434e278e7e8d8ae37e05b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:02.398284 16933 tablet_service.cc:1467] Processing CreateTablet for tablet de8be0f60d434e278e7e8d8ae37e05b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55d3d8857b6c4c93a71fe1c1b65d0327]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:53:02.398814 16933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de8be0f60d434e278e7e8d8ae37e05b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:02.398984 17073 tablet_service.cc:1467] Processing CreateTablet for tablet de8be0f60d434e278e7e8d8ae37e05b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55d3d8857b6c4c93a71fe1c1b65d0327]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:53:02.399530 17073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de8be0f60d434e278e7e8d8ae37e05b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:02.405206 17309 tablet_bootstrap.cc:492] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8: Bootstrap starting.
I20260926 20:53:02.405514 17308 tablet_bootstrap.cc:492] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Bootstrap starting.
I20260926 20:53:02.405570 17307 tablet_bootstrap.cc:492] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188: Bootstrap starting.
I20260926 20:53:02.406776 17309 tablet_bootstrap.cc:654] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:02.407192 17307 tablet_bootstrap.cc:654] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:02.407192 17308 tablet_bootstrap.cc:654] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:02.409602 17308 tablet_bootstrap.cc:492] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd: No bootstrap required, opened a new log
I20260926 20:53:02.409809 17308 ts_tablet_manager.cc:1397] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 20:53:02.410326 17309 tablet_bootstrap.cc:492] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8: No bootstrap required, opened a new log
I20260926 20:53:02.410524 17309 ts_tablet_manager.cc:1397] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260926 20:53:02.410465 17308 raft_consensus.cc:348] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.410719 17308 raft_consensus.cc:374] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:02.410830 17308 raft_consensus.cc:729] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a90f39bc7eb4caea5d5d62edb3a52fd, State: Initialized, Role: FOLLOWER
I20260926 20:53:02.411232 17307 tablet_bootstrap.cc:492] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188: No bootstrap required, opened a new log
I20260926 20:53:02.411106 17308 consensus_queue.cc:260] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.411386 17307 ts_tablet_manager.cc:1397] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:53:02.411237 17309 raft_consensus.cc:348] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.411507 17309 raft_consensus.cc:374] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:02.411609 17309 raft_consensus.cc:729] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d34df039554a42acc3bd191ee23bb8, State: Initialized, Role: FOLLOWER
I20260926 20:53:02.411548 17314 tablet_replica.cc:439] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } }
I20260926 20:53:02.411831 17314 tablet_replica.cc:442] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:53:02.411840 17309 consensus_queue.cc:260] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.412276 17315 tablet_replica.cc:439] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } }
I20260926 20:53:02.412490 17309 ts_tablet_manager.cc:1428] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 20:53:02.412577 17315 tablet_replica.cc:442] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:53:02.413236 17308 ts_tablet_manager.cc:1428] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 20:53:02.412127 17307 raft_consensus.cc:348] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.413729 17307 raft_consensus.cc:374] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:02.413823 17307 raft_consensus.cc:729] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adaf6826ac464b79af323870bd2ca188, State: Initialized, Role: FOLLOWER
I20260926 20:53:02.414063 17307 consensus_queue.cc:260] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.414541 17313 tablet_replica.cc:439] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } }
I20260926 20:53:02.414876 17313 tablet_replica.cc:442] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:53:02.415926 17307 ts_tablet_manager.cc:1428] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:53:02.615731 17314 raft_consensus.cc:482] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:02.616000 17314 raft_consensus.cc:504] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.616627 17314 leader_election.cc:288] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130:45857), adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937)
I20260926 20:53:02.617321 17097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8be0f60d434e278e7e8d8ae37e05b2" candidate_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d34df039554a42acc3bd191ee23bb8" is_pre_election: true
I20260926 20:53:02.617405 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8be0f60d434e278e7e8d8ae37e05b2" candidate_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adaf6826ac464b79af323870bd2ca188" is_pre_election: true
I20260926 20:53:02.617652 17097 raft_consensus.cc:2454] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a90f39bc7eb4caea5d5d62edb3a52fd in term 0.
I20260926 20:53:02.617703 17234 raft_consensus.cc:2454] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a90f39bc7eb4caea5d5d62edb3a52fd in term 0.
I20260926 20:53:02.618116 16887 leader_election.cc:302] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a90f39bc7eb4caea5d5d62edb3a52fd, 32d34df039554a42acc3bd191ee23bb8; no voters:
I20260926 20:53:02.618399 17314 raft_consensus.cc:2781] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:53:02.618510 17314 raft_consensus.cc:482] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:53:02.618626 17314 raft_consensus.cc:3037] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:02.620661 17314 raft_consensus.cc:504] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.621275 17314 leader_election.cc:288] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [CANDIDATE]: Term 1 election: Requested vote from peers 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130:45857), adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937)
I20260926 20:53:02.621717 17097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8be0f60d434e278e7e8d8ae37e05b2" candidate_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d34df039554a42acc3bd191ee23bb8"
I20260926 20:53:02.621775 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de8be0f60d434e278e7e8d8ae37e05b2" candidate_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adaf6826ac464b79af323870bd2ca188"
I20260926 20:53:02.621996 17234 raft_consensus.cc:3037] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:02.621977 17097 raft_consensus.cc:3037] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:02.624260 17234 raft_consensus.cc:2454] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a90f39bc7eb4caea5d5d62edb3a52fd in term 1.
I20260926 20:53:02.624384 17097 raft_consensus.cc:2454] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a90f39bc7eb4caea5d5d62edb3a52fd in term 1.
I20260926 20:53:02.624681 16887 leader_election.cc:302] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a90f39bc7eb4caea5d5d62edb3a52fd, adaf6826ac464b79af323870bd2ca188; no voters:
I20260926 20:53:02.624972 17314 raft_consensus.cc:2781] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:02.625430 17314 raft_consensus.cc:686] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 LEADER]: Becoming Leader. State: Replica: 2a90f39bc7eb4caea5d5d62edb3a52fd, State: Running, Role: LEADER
I20260926 20:53:02.625874 17314 consensus_queue.cc:237] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.627862 17355 tablet_replica.cc:439] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd: TxnStatusTablet state changed. Reason: New leader 2a90f39bc7eb4caea5d5d62edb3a52fd. Latest consensus state: current_term: 1 leader_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } }
I20260926 20:53:02.628278 17355 tablet_replica.cc:442] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:53:02.628917 17357 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:53:02.629130 17357 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:53:02.629755 16796 catalog_manager.cc:5495] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a90f39bc7eb4caea5d5d62edb3a52fd (127.8.249.129). New cstate: current_term: 1 leader_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } health_report { overall_health: UNKNOWN } } }
I20260926 20:53:02.652784 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.646386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.8.249.129:42749)
0926 20:53:02.646555 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:02.646559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:02.646580 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:53:02.646896 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:02.646905 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:02.647029 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:02.647194 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:02.647580 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.648777 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.650064 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.650743 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.650903 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.651341 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:02.652137 (+ 796us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:02.652280 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:02.652393 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 20:53:02.652479 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:53:02.652786 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.646257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:42749 (local address 127.0.0.1:38312)
0926 20:53:02.646491 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:02.646513 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:02.646644 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:02.647228 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:02.647232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:02.647242 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:53:02.647441 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.647448 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.648930 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.648933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.649923 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.649933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.650925 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.650930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.651082 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.651153 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:53:02.652257 (+ 1104us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:53:02.652276 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:53:02.652351 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:53:02.652469 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 20:53:02.659106 17234 raft_consensus.cc:1261] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Refusing update from remote peer 2a90f39bc7eb4caea5d5d62edb3a52fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:02.659107 17097 raft_consensus.cc:1261] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Refusing update from remote peer 2a90f39bc7eb4caea5d5d62edb3a52fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:02.659952 17355 consensus_queue.cc:1035] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:02.660738 17314 consensus_queue.cc:1035] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:02.669359 17313 tablet_replica.cc:439] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188: TxnStatusTablet state changed. Reason: New leader 2a90f39bc7eb4caea5d5d62edb3a52fd. Latest consensus state: current_term: 1 leader_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } }
I20260926 20:53:02.669694 17313 tablet_replica.cc:442] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:53:02.670477 17315 tablet_replica.cc:439] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8: TxnStatusTablet state changed. Reason: New leader 2a90f39bc7eb4caea5d5d62edb3a52fd. Latest consensus state: current_term: 1 leader_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } }
I20260926 20:53:02.670814 17315 tablet_replica.cc:442] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:53:02.673676 17362 mvcc.cc:205] Tried to move back new op lower bound from 7333707704959393792 to 7333707704839593984. Current Snapshot: MvccSnapshot[applied={T|T < 7333707704959393792}]
I20260926 20:53:02.681941 17313 tablet_replica.cc:439] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } }
I20260926 20:53:02.682263 17313 tablet_replica.cc:442] T de8be0f60d434e278e7e8d8ae37e05b2 P adaf6826ac464b79af323870bd2ca188: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:53:02.686416 17315 tablet_replica.cc:439] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } }
I20260926 20:53:02.686710 17315 tablet_replica.cc:442] T de8be0f60d434e278e7e8d8ae37e05b2 P 32d34df039554a42acc3bd191ee23bb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:53:02.688477 17355 tablet_replica.cc:439] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } }
I20260926 20:53:02.688822 17355 tablet_replica.cc:442] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:53:02.688560 17365 tablet_replica.cc:439] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } } }
I20260926 20:53:02.688997 17365 tablet_replica.cc:442] T de8be0f60d434e278e7e8d8ae37e05b2 P 2a90f39bc7eb4caea5d5d62edb3a52fd: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:53:02.699546 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.693106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.8.249.130:45857)
0926 20:53:02.693309 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:02.693316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:02.693647 (+ 331us) server_negotiation.cc:407] Connection header received
0926 20:53:02.693867 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:02.693871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:02.693946 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:02.694072 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:02.694589 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.695402 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.697374 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.697754 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.697858 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.698344 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:02.698972 (+ 628us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:02.699077 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:02.699213 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 20:53:02.699279 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:53:02.719524 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.711390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:42749 (local address 127.8.249.130:38013)
0926 20:53:02.712116 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:02.712143 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:02.712313 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:02.712730 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:02.712735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:02.712748 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:53:02.713244 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.713261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.714510 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.714518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.716362 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.716380 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.717932 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.717940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.718127 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.718224 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:53:02.718920 (+ 696us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:53:02.718938 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 20:53:02.719028 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:53:02.719174 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":372,"threads_started":1}
I20260926 20:53:02.719532 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.711379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:38013 (local address 127.8.249.129:42749)
0926 20:53:02.711600 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:02.711609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:02.712266 (+ 657us) server_negotiation.cc:407] Connection header received
0926 20:53:02.712499 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:02.712503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:02.712597 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:02.712690 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:02.713426 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.714365 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.716557 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.717480 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.717636 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.718523 (+ 887us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:02.718686 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:02.718830 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:02.719081 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 20:53:02.719173 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:53:02.730446 17234 raft_consensus.cc:1261] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Refusing update from remote peer 32d34df039554a42acc3bd191ee23bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:02.730540 16954 raft_consensus.cc:1261] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Refusing update from remote peer 32d34df039554a42acc3bd191ee23bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:02.731243 17343 consensus_queue.cc:1035] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:02.731593 17315 consensus_queue.cc:1035] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:02.767398 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.760463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:45857 (local address 127.8.249.129:52593)
0926 20:53:02.761134 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:02.761160 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:02.761301 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:02.761750 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:02.761756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:02.761770 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:53:02.762239 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.762258 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.763798 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.763802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.765148 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:02.765163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.766064 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.766074 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:02.766248 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.766342 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:53:02.766881 (+ 539us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:53:02.766904 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:53:02.766974 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:53:02.767091 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":279,"threads_started":1}
I20260926 20:53:02.767387 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.760586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:52593 (local address 127.8.249.130:45857)
0926 20:53:02.760735 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:02.760740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:02.761251 (+ 511us) server_negotiation.cc:407] Connection header received
0926 20:53:02.761508 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:02.761513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:02.761616 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:02.761716 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:02.762411 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.763661 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.765343 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.765867 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.766004 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.766587 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:02.766734 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:02.766889 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:02.767009 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:53:02.767090 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 20:53:02.806777 16795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:02.807065 16795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:02.810432 16795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585: Sending DeleteTablet for 3 replicas of tablet 8283f6a0765e431eb29a52582c0731f7
I20260926 20:53:02.811225 16933 tablet_service.cc:1514] Processing DeleteTablet for tablet 8283f6a0765e431eb29a52582c0731f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:02 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:38310
I20260926 20:53:02.811470 17073 tablet_service.cc:1514] Processing DeleteTablet for tablet 8283f6a0765e431eb29a52582c0731f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:02 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:38836
I20260926 20:53:02.812569 17213 tablet_service.cc:1514] Processing DeleteTablet for tablet 8283f6a0765e431eb29a52582c0731f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:02 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:46698
I20260926 20:53:02.826568 17381 tablet_replica.cc:330] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd: stopping tablet replica
I20260926 20:53:02.826853 17381 raft_consensus.cc:2229] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:02.827172 17381 raft_consensus.cc:2258] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:02.828357 17381 ts_tablet_manager.cc:1905] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:02.831629 17382 tablet_replica.cc:330] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8: stopping tablet replica
I20260926 20:53:02.831815 17382 raft_consensus.cc:2229] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:53:02.832307 17382 raft_consensus.cc:2258] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:02.832908 17383 tablet_replica.cc:330] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188: stopping tablet replica
I20260926 20:53:02.833122 17383 raft_consensus.cc:2229] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:02.833357 17383 raft_consensus.cc:2258] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:02.835486 17383 ts_tablet_manager.cc:1905] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:02.836074 17381 ts_tablet_manager.cc:1918] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:53:02.836215 17381 log.cc:1196] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/8283f6a0765e431eb29a52582c0731f7
I20260926 20:53:02.836690 17381 ts_tablet_manager.cc:1939] T 8283f6a0765e431eb29a52582c0731f7 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Deleting consensus metadata
I20260926 20:53:02.837932 16781 catalog_manager.cc:4841] TS 2a90f39bc7eb4caea5d5d62edb3a52fd (127.8.249.129:42749): tablet 8283f6a0765e431eb29a52582c0731f7 (table test-table [id=d4d39100be89420dac5cf208519c675f]) successfully deleted
I20260926 20:53:02.838209 17382 ts_tablet_manager.cc:1905] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:02.845635 17382 ts_tablet_manager.cc:1918] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:53:02.845801 17382 log.cc:1196] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/8283f6a0765e431eb29a52582c0731f7
I20260926 20:53:02.846304 17382 ts_tablet_manager.cc:1939] T 8283f6a0765e431eb29a52582c0731f7 P 32d34df039554a42acc3bd191ee23bb8: Deleting consensus metadata
I20260926 20:53:02.847635 16784 catalog_manager.cc:4841] TS 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130:45857): tablet 8283f6a0765e431eb29a52582c0731f7 (table test-table [id=d4d39100be89420dac5cf208519c675f]) successfully deleted
I20260926 20:53:02.853660 17383 ts_tablet_manager.cc:1918] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:53:02.853801 17383 log.cc:1196] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/8283f6a0765e431eb29a52582c0731f7
I20260926 20:53:02.854233 17383 ts_tablet_manager.cc:1939] T 8283f6a0765e431eb29a52582c0731f7 P adaf6826ac464b79af323870bd2ca188: Deleting consensus metadata
I20260926 20:53:02.855417 16784 catalog_manager.cc:4841] TS adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937): tablet 8283f6a0765e431eb29a52582c0731f7 (table test-table [id=d4d39100be89420dac5cf208519c675f]) successfully deleted
Sep 26 20:53:02 dist-test-slave-2dg3 krb5kdc[16752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:53:02 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:02 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:53:02.930917 17390 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:02.900802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.8.249.190:40349)
0926 20:53:02.916439 (+ 15637us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:02.916447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:02.917056 (+ 609us) server_negotiation.cc:407] Connection header received
0926 20:53:02.917255 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:02.917262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:02.917371 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:02.917513 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:02.919372 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.920618 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.922641 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.923368 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.923518 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.930573 (+ 7055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36616: BlockingRecv error: recv got EOF from 127.0.0.1:36616 (error 108)
Metrics: {"server-negotiator.queue_time_us":15452,"thread_start_us":16412,"threads_started":1}
Sep 26 20:53:02 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM
I20260926 20:53:02.955001 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:02.936470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.8.249.190:40349)
0926 20:53:02.936648 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:02.936655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:02.937115 (+ 460us) server_negotiation.cc:407] Connection header received
0926 20:53:02.937405 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:02.937411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:02.937507 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:02.937619 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:02.939227 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.940360 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.942246 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:02.942982 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:02.943115 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:02.948315 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:02.948322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:02.948327 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:02.948372 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:02.951948 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:02.952765 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:02.952772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:02.952776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:02.952943 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:02.953377 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:02.953382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:02.953386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:02.953757 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:02.954047 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:02.954337 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 20:53:02.954582 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:53:02.959255 16795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:53:02.960037 16795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:53:02.968003 16933 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6b4e4062ab4ac68695694813cca1ca (DEFAULT_TABLE table=test-table [id=333f9d6bf1e84dc996323784deacb6f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:02.968356 17213 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6b4e4062ab4ac68695694813cca1ca (DEFAULT_TABLE table=test-table [id=333f9d6bf1e84dc996323784deacb6f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:02.968570 16933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6b4e4062ab4ac68695694813cca1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:02.968751 17213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6b4e4062ab4ac68695694813cca1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:02.975548 17073 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6b4e4062ab4ac68695694813cca1ca (DEFAULT_TABLE table=test-table [id=333f9d6bf1e84dc996323784deacb6f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:02.976037 17073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6b4e4062ab4ac68695694813cca1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:02.981647 17397 tablet_bootstrap.cc:492] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188: Bootstrap starting.
I20260926 20:53:02.983979 17397 tablet_bootstrap.cc:654] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:02.992239 17397 tablet_bootstrap.cc:492] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188: No bootstrap required, opened a new log
I20260926 20:53:02.992404 17397 ts_tablet_manager.cc:1397] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260926 20:53:02.993176 17397 raft_consensus.cc:348] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.993456 17397 raft_consensus.cc:374] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:02.993546 17397 raft_consensus.cc:729] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adaf6826ac464b79af323870bd2ca188, State: Initialized, Role: FOLLOWER
I20260926 20:53:02.993805 17397 consensus_queue.cc:260] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:02.995343 17399 tablet_bootstrap.cc:492] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8: Bootstrap starting.
I20260926 20:53:02.996737 17397 ts_tablet_manager.cc:1428] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260926 20:53:02.997874 17399 tablet_bootstrap.cc:654] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:02.999310 17400 tablet_bootstrap.cc:492] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd: Bootstrap starting.
I20260926 20:53:03.001116 17400 tablet_bootstrap.cc:654] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:03.006780 17400 tablet_bootstrap.cc:492] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd: No bootstrap required, opened a new log
I20260926 20:53:03.006982 17400 ts_tablet_manager.cc:1397] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260926 20:53:03.008147 17399 tablet_bootstrap.cc:492] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8: No bootstrap required, opened a new log
I20260926 20:53:03.007982 17400 raft_consensus.cc:348] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.008278 17400 raft_consensus.cc:374] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:03.008386 17400 raft_consensus.cc:729] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a90f39bc7eb4caea5d5d62edb3a52fd, State: Initialized, Role: FOLLOWER
I20260926 20:53:03.008447 17399 ts_tablet_manager.cc:1397] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260926 20:53:03.008684 17400 consensus_queue.cc:260] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.009992 17400 ts_tablet_manager.cc:1428] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:53:03.010600 17399 raft_consensus.cc:348] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.010968 17399 raft_consensus.cc:374] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:03.011133 17399 raft_consensus.cc:729] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d34df039554a42acc3bd191ee23bb8, State: Initialized, Role: FOLLOWER
I20260926 20:53:03.011948 17399 consensus_queue.cc:260] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.012627 17399 ts_tablet_manager.cc:1428] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:53:03.230396 17365 raft_consensus.cc:482] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:03.230618 17365 raft_consensus.cc:504] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.231391 17365 leader_election.cc:288] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130:45857), adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937)
I20260926 20:53:03.232064 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6b4e4062ab4ac68695694813cca1ca" candidate_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adaf6826ac464b79af323870bd2ca188" is_pre_election: true
I20260926 20:53:03.232087 17097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6b4e4062ab4ac68695694813cca1ca" candidate_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d34df039554a42acc3bd191ee23bb8" is_pre_election: true
I20260926 20:53:03.232324 17234 raft_consensus.cc:2454] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a90f39bc7eb4caea5d5d62edb3a52fd in term 0.
I20260926 20:53:03.232410 17097 raft_consensus.cc:2454] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a90f39bc7eb4caea5d5d62edb3a52fd in term 0.
I20260926 20:53:03.232803 16887 leader_election.cc:302] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a90f39bc7eb4caea5d5d62edb3a52fd, adaf6826ac464b79af323870bd2ca188; no voters:
I20260926 20:53:03.233083 17365 raft_consensus.cc:2781] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:53:03.233215 17365 raft_consensus.cc:482] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:53:03.233309 17365 raft_consensus.cc:3037] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:03.235570 17365 raft_consensus.cc:504] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.236155 17365 leader_election.cc:288] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [CANDIDATE]: Term 1 election: Requested vote from peers 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130:45857), adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937)
I20260926 20:53:03.236701 17097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6b4e4062ab4ac68695694813cca1ca" candidate_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d34df039554a42acc3bd191ee23bb8"
I20260926 20:53:03.236837 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6b4e4062ab4ac68695694813cca1ca" candidate_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adaf6826ac464b79af323870bd2ca188"
I20260926 20:53:03.236956 17097 raft_consensus.cc:3037] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:03.237083 17234 raft_consensus.cc:3037] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:03.239432 17097 raft_consensus.cc:2454] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a90f39bc7eb4caea5d5d62edb3a52fd in term 1.
I20260926 20:53:03.239889 16887 leader_election.cc:302] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a90f39bc7eb4caea5d5d62edb3a52fd, 32d34df039554a42acc3bd191ee23bb8; no voters:
I20260926 20:53:03.240218 17365 raft_consensus.cc:2781] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:03.240228 17234 raft_consensus.cc:2454] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a90f39bc7eb4caea5d5d62edb3a52fd in term 1.
I20260926 20:53:03.240414 17365 raft_consensus.cc:686] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 LEADER]: Becoming Leader. State: Replica: 2a90f39bc7eb4caea5d5d62edb3a52fd, State: Running, Role: LEADER
I20260926 20:53:03.240892 17365 consensus_queue.cc:237] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.243904 16796 catalog_manager.cc:5495] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a90f39bc7eb4caea5d5d62edb3a52fd (127.8.249.129). New cstate: current_term: 1 leader_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } health_report { overall_health: UNKNOWN } } }
I20260926 20:53:03.293366 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:03.285813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.8.249.129:42749)
0926 20:53:03.286603 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:03.286609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:03.286629 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:53:03.286694 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:03.286699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:03.286774 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:03.287332 (+ 558us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:03.287491 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:03.288676 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:03.290651 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:03.291337 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:03.291459 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:03.292003 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:03.292154 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:03.292251 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:03.292412 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 20:53:03.292484 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":553,"threads_started":1}
I20260926 20:53:03.296774 17097 raft_consensus.cc:1261] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Refusing update from remote peer 2a90f39bc7eb4caea5d5d62edb3a52fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:03.296808 17234 raft_consensus.cc:1261] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Refusing update from remote peer 2a90f39bc7eb4caea5d5d62edb3a52fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:03.297338 17365 consensus_queue.cc:1035] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:03.297643 17355 consensus_queue.cc:1035] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:03.306502 17406 mvcc.cc:205] Tried to move back new op lower bound from 7333707707578310656 to 7333707707357134848. Current Snapshot: MvccSnapshot[applied={T|T < 7333707707578310656}]
I20260926 20:53:03.327936 16796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:03.328209 16796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:03.331521 16796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585: Sending DeleteTablet for 3 replicas of tablet 3e6b4e4062ab4ac68695694813cca1ca
I20260926 20:53:03.332289 16933 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6b4e4062ab4ac68695694813cca1ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:03 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:38310
I20260926 20:53:03.332546 17381 tablet_replica.cc:330] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd: stopping tablet replica
I20260926 20:53:03.332753 17381 raft_consensus.cc:2229] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:53:03.332804 17073 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6b4e4062ab4ac68695694813cca1ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:03 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:38836
I20260926 20:53:03.333083 17213 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6b4e4062ab4ac68695694813cca1ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:03 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:46698
I20260926 20:53:03.333178 17381 raft_consensus.cc:2258] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:03.333336 17382 tablet_replica.cc:330] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8: stopping tablet replica
I20260926 20:53:03.333540 17382 raft_consensus.cc:2229] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:03.333747 17382 raft_consensus.cc:2258] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:03.333972 17383 tablet_replica.cc:330] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188: stopping tablet replica
I20260926 20:53:03.333858 16796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:53:03.334146 17383 raft_consensus.cc:2229] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:03.334178 17381 ts_tablet_manager.cc:1905] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:03.334328 17383 raft_consensus.cc:2258] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 20:53:03.334717 16796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:53:03.335744 17383 ts_tablet_manager.cc:1905] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:03.340601 17381 ts_tablet_manager.cc:1918] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:03.340745 17381 log.cc:1196] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/3e6b4e4062ab4ac68695694813cca1ca
I20260926 20:53:03.341171 17381 ts_tablet_manager.cc:1939] T 3e6b4e4062ab4ac68695694813cca1ca P 2a90f39bc7eb4caea5d5d62edb3a52fd: Deleting consensus metadata
I20260926 20:53:03.342096 17213 tablet_service.cc:1467] Processing CreateTablet for tablet ef75465aefd848d4b918f775f8e16014 (DEFAULT_TABLE table=test-table [id=30fdb6e073404071969a19d524bae8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:03.342510 17213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef75465aefd848d4b918f775f8e16014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:03.343151 16933 tablet_service.cc:1467] Processing CreateTablet for tablet ef75465aefd848d4b918f775f8e16014 (DEFAULT_TABLE table=test-table [id=30fdb6e073404071969a19d524bae8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:03.343538 16933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef75465aefd848d4b918f775f8e16014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:03.345777 17383 ts_tablet_manager.cc:1918] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:03.345973 17383 log.cc:1196] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/3e6b4e4062ab4ac68695694813cca1ca
I20260926 20:53:03.346277 17073 tablet_service.cc:1467] Processing CreateTablet for tablet ef75465aefd848d4b918f775f8e16014 (DEFAULT_TABLE table=test-table [id=30fdb6e073404071969a19d524bae8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:03.346691 17073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef75465aefd848d4b918f775f8e16014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:03.346532 17383 ts_tablet_manager.cc:1939] T 3e6b4e4062ab4ac68695694813cca1ca P adaf6826ac464b79af323870bd2ca188: Deleting consensus metadata
I20260926 20:53:03.350440 16784 catalog_manager.cc:4841] TS adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937): tablet 3e6b4e4062ab4ac68695694813cca1ca (table test-table [id=333f9d6bf1e84dc996323784deacb6f4]) successfully deleted
I20260926 20:53:03.350929 16781 catalog_manager.cc:4841] TS 2a90f39bc7eb4caea5d5d62edb3a52fd (127.8.249.129:42749): tablet 3e6b4e4062ab4ac68695694813cca1ca (table test-table [id=333f9d6bf1e84dc996323784deacb6f4]) successfully deleted
I20260926 20:53:03.351343 17382 ts_tablet_manager.cc:1905] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:03.360870 17397 tablet_bootstrap.cc:492] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188: Bootstrap starting.
I20260926 20:53:03.362486 17397 tablet_bootstrap.cc:654] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:03.365417 17382 ts_tablet_manager.cc:1918] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:03.368832 17400 tablet_bootstrap.cc:492] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Bootstrap starting.
I20260926 20:53:03.369027 17382 log.cc:1196] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/3e6b4e4062ab4ac68695694813cca1ca
I20260926 20:53:03.369498 17382 ts_tablet_manager.cc:1939] T 3e6b4e4062ab4ac68695694813cca1ca P 32d34df039554a42acc3bd191ee23bb8: Deleting consensus metadata
I20260926 20:53:03.370609 16784 catalog_manager.cc:4841] TS 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130:45857): tablet 3e6b4e4062ab4ac68695694813cca1ca (table test-table [id=333f9d6bf1e84dc996323784deacb6f4]) successfully deleted
I20260926 20:53:03.371016 17400 tablet_bootstrap.cc:654] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:03.368240 17399 tablet_bootstrap.cc:492] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8: Bootstrap starting.
I20260926 20:53:03.371877 17397 tablet_bootstrap.cc:492] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188: No bootstrap required, opened a new log
I20260926 20:53:03.372153 17397 ts_tablet_manager.cc:1397] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260926 20:53:03.372857 17399 tablet_bootstrap.cc:654] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:03.373310 17397 raft_consensus.cc:348] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.373754 17397 raft_consensus.cc:374] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:03.373895 17397 raft_consensus.cc:729] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adaf6826ac464b79af323870bd2ca188, State: Initialized, Role: FOLLOWER
I20260926 20:53:03.375548 17397 consensus_queue.cc:260] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.379666 17397 ts_tablet_manager.cc:1428] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260926 20:53:03.385083 17400 tablet_bootstrap.cc:492] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd: No bootstrap required, opened a new log
I20260926 20:53:03.385087 17399 tablet_bootstrap.cc:492] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8: No bootstrap required, opened a new log
I20260926 20:53:03.385267 17399 ts_tablet_manager.cc:1397] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20260926 20:53:03.385274 17400 ts_tablet_manager.cc:1397] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260926 20:53:03.386060 17399 raft_consensus.cc:348] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.386073 17400 raft_consensus.cc:348] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.386315 17399 raft_consensus.cc:374] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:03.386324 17400 raft_consensus.cc:374] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:03.386394 17399 raft_consensus.cc:729] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d34df039554a42acc3bd191ee23bb8, State: Initialized, Role: FOLLOWER
I20260926 20:53:03.386405 17400 raft_consensus.cc:729] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a90f39bc7eb4caea5d5d62edb3a52fd, State: Initialized, Role: FOLLOWER
I20260926 20:53:03.386672 17400 consensus_queue.cc:260] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.386641 17399 consensus_queue.cc:260] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.388386 17400 ts_tablet_manager.cc:1428] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:53:03.388517 17378 raft_consensus.cc:482] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:03.388674 17378 raft_consensus.cc:504] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.389140 17399 ts_tablet_manager.cc:1428] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:53:03.389425 17378 leader_election.cc:288] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a90f39bc7eb4caea5d5d62edb3a52fd (127.8.249.129:42749), adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937)
I20260926 20:53:03.389880 16954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef75465aefd848d4b918f775f8e16014" candidate_uuid: "32d34df039554a42acc3bd191ee23bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" is_pre_election: true
I20260926 20:53:03.390142 16954 raft_consensus.cc:2454] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32d34df039554a42acc3bd191ee23bb8 in term 0.
I20260926 20:53:03.390076 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef75465aefd848d4b918f775f8e16014" candidate_uuid: "32d34df039554a42acc3bd191ee23bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adaf6826ac464b79af323870bd2ca188" is_pre_election: true
I20260926 20:53:03.390307 17234 raft_consensus.cc:2454] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32d34df039554a42acc3bd191ee23bb8 in term 0.
I20260926 20:53:03.390599 17024 leader_election.cc:302] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a90f39bc7eb4caea5d5d62edb3a52fd, 32d34df039554a42acc3bd191ee23bb8; no voters:
I20260926 20:53:03.391083 17378 raft_consensus.cc:2781] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:53:03.391227 17378 raft_consensus.cc:482] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:53:03.391317 17378 raft_consensus.cc:3037] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:03.393656 17378 raft_consensus.cc:504] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.394681 16954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef75465aefd848d4b918f775f8e16014" candidate_uuid: "32d34df039554a42acc3bd191ee23bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd"
I20260926 20:53:03.394929 16954 raft_consensus.cc:3037] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:03.395061 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef75465aefd848d4b918f775f8e16014" candidate_uuid: "32d34df039554a42acc3bd191ee23bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adaf6826ac464b79af323870bd2ca188"
I20260926 20:53:03.395279 17234 raft_consensus.cc:3037] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:03.398092 17234 raft_consensus.cc:2454] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32d34df039554a42acc3bd191ee23bb8 in term 1.
I20260926 20:53:03.398092 16954 raft_consensus.cc:2454] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32d34df039554a42acc3bd191ee23bb8 in term 1.
I20260926 20:53:03.394444 17378 leader_election.cc:288] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [CANDIDATE]: Term 1 election: Requested vote from peers 2a90f39bc7eb4caea5d5d62edb3a52fd (127.8.249.129:42749), adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937)
I20260926 20:53:03.398458 17024 leader_election.cc:302] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a90f39bc7eb4caea5d5d62edb3a52fd, 32d34df039554a42acc3bd191ee23bb8; no voters:
I20260926 20:53:03.398829 17378 raft_consensus.cc:2781] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:03.399305 17378 raft_consensus.cc:686] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [term 1 LEADER]: Becoming Leader. State: Replica: 32d34df039554a42acc3bd191ee23bb8, State: Running, Role: LEADER
I20260926 20:53:03.399605 17378 consensus_queue.cc:237] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } }
I20260926 20:53:03.402235 16795 catalog_manager.cc:5495] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130). New cstate: current_term: 1 leader_uuid: "32d34df039554a42acc3bd191ee23bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32d34df039554a42acc3bd191ee23bb8" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 } health_report { overall_health: UNKNOWN } } }
I20260926 20:53:03.426997 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:03.420349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.8.249.130:45857)
0926 20:53:03.420966 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:03.420970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:03.420997 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:53:03.421053 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:03.421057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:03.421119 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:03.421240 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:03.421726 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:03.422957 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:03.424657 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:03.425322 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:03.425448 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:03.426106 (+ 658us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:03.426251 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:03.426353 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:03.426500 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 20:53:03.426580 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":290,"threads_started":1}
I20260926 20:53:03.437036 17234 raft_consensus.cc:1261] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Refusing update from remote peer 32d34df039554a42acc3bd191ee23bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:03.437080 16954 raft_consensus.cc:1261] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Refusing update from remote peer 32d34df039554a42acc3bd191ee23bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:03.437651 17418 consensus_queue.cc:1035] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adaf6826ac464b79af323870bd2ca188" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:03.438002 17378 consensus_queue.cc:1035] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a90f39bc7eb4caea5d5d62edb3a52fd" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:03.494696 16795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:03.494989 16795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:03.497843 16795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b221558242e242b6af9bf7d678e94585: Sending DeleteTablet for 3 replicas of tablet ef75465aefd848d4b918f775f8e16014
I20260926 20:53:03.498636 16933 tablet_service.cc:1514] Processing DeleteTablet for tablet ef75465aefd848d4b918f775f8e16014 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:03 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:38310
I20260926 20:53:03.498754 17073 tablet_service.cc:1514] Processing DeleteTablet for tablet ef75465aefd848d4b918f775f8e16014 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:03 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:38836
I20260926 20:53:03.498829 17213 tablet_service.cc:1514] Processing DeleteTablet for tablet ef75465aefd848d4b918f775f8e16014 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:03 UTC) from {username='oryx', principal='oryx/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:46698
I20260926 20:53:03.498970 17381 tablet_replica.cc:330] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd: stopping tablet replica
I20260926 20:53:03.499051 17382 tablet_replica.cc:330] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8: stopping tablet replica
I20260926 20:53:03.499140 17381 raft_consensus.cc:2229] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:03.499251 17382 raft_consensus.cc:2229] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:53:03.499298 17381 raft_consensus.cc:2258] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:03.499565 17382 raft_consensus.cc:2258] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:03.499840 17383 tablet_replica.cc:330] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188: stopping tablet replica
I20260926 20:53:03.500005 17383 raft_consensus.cc:2229] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:03.500191 17383 raft_consensus.cc:2258] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:03.500298 17381 ts_tablet_manager.cc:1905] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:03.501477 17383 ts_tablet_manager.cc:1905] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:03.505525 17381 ts_tablet_manager.cc:1918] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:53:03.505669 17381 log.cc:1196] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/ef75465aefd848d4b918f775f8e16014
I20260926 20:53:03.505712 17382 ts_tablet_manager.cc:1905] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:03.506006 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 16868
I20260926 20:53:03.506134 17381 ts_tablet_manager.cc:1939] T ef75465aefd848d4b918f775f8e16014 P 2a90f39bc7eb4caea5d5d62edb3a52fd: Deleting consensus metadata
I20260926 20:53:03.511255 16781 catalog_manager.cc:4841] TS 2a90f39bc7eb4caea5d5d62edb3a52fd (127.8.249.129:42749): tablet ef75465aefd848d4b918f775f8e16014 (table test-table [id=30fdb6e073404071969a19d524bae8a1]) successfully deleted
I20260926 20:53:03.512611 17383 ts_tablet_manager.cc:1918] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:53:03.512763 17383 log.cc:1196] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/ef75465aefd848d4b918f775f8e16014
I20260926 20:53:03.512682 17382 ts_tablet_manager.cc:1918] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:53:03.513110 17382 log.cc:1196] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/ef75465aefd848d4b918f775f8e16014
I20260926 20:53:03.513178 17383 ts_tablet_manager.cc:1939] T ef75465aefd848d4b918f775f8e16014 P adaf6826ac464b79af323870bd2ca188: Deleting consensus metadata
I20260926 20:53:03.513624 17382 ts_tablet_manager.cc:1939] T ef75465aefd848d4b918f775f8e16014 P 32d34df039554a42acc3bd191ee23bb8: Deleting consensus metadata
I20260926 20:53:03.514243 16784 catalog_manager.cc:4841] TS adaf6826ac464b79af323870bd2ca188 (127.8.249.131:36937): tablet ef75465aefd848d4b918f775f8e16014 (table test-table [id=30fdb6e073404071969a19d524bae8a1]) successfully deleted
I20260926 20:53:03.514760 16784 catalog_manager.cc:4841] TS 32d34df039554a42acc3bd191ee23bb8 (127.8.249.130:45857): tablet ef75465aefd848d4b918f775f8e16014 (table test-table [id=30fdb6e073404071969a19d524bae8a1]) successfully deleted
Sep 26 20:53:03 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.249.129@KRBTEST.COM
I20260926 20:53:03.525249 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:03.507630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.8.249.129:42749)
0926 20:53:03.508136 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:03.508142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:03.508168 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:53:03.508233 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:03.508238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:03.508309 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:03.508409 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:03.510043 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:03.511192 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:03.513021 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:03.513727 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:03.513850 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:03.518146 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:03.518164 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:03.518186 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:03.518226 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:03.522283 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:03.523085 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:03.523096 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:03.523101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:03.523283 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:03.523671 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:03.523677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:03.523681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:03.524123 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:03.524442 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:03.524621 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:53:03.524866 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397}
I20260926 20:53:03.525990 16999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:03.670771 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 16868
I20260926 20:53:03.695077 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 17008
Sep 26 20:53:03 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.249.130@KRBTEST.COM
I20260926 20:53:03.709949 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:03.695923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.8.249.130:45857)
0926 20:53:03.696090 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:03.696095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:03.696111 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:53:03.696249 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:03.696253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:03.696323 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:03.696410 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:03.697664 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:03.698399 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:03.699820 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:03.700228 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:03.700333 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:03.704545 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:03.704566 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:03.704582 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:03.704614 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:03.707759 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:03.708317 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:03.708323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:03.708326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:03.708451 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:03.708753 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:03.708756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:03.708758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:03.709103 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:03.709325 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:03.709540 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 20:53:03.709687 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:53:03.710573 17138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:03.835845 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 17008
I20260926 20:53:03.858382 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 17147
Sep 26 20:53:03 dist-test-slave-2dg3 krb5kdc[16752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.249.131@KRBTEST.COM
I20260926 20:53:03.873504 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:03.859178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.8.249.131:36937)
0926 20:53:03.859636 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:03.859640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:03.859656 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:53:03.859709 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:03.859713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:03.859773 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:03.859869 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:03.861075 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:03.861812 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:03.863164 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:03.863586 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:03.863686 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:03.867942 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:03.867956 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:03.867972 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:03.868004 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:03.870625 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:03.871384 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:03.871400 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:03.871403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:03.871545 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:03.871932 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:03.871938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:03.871946 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:03.872323 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:03.872637 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:03.872936 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 20:53:03.873163 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":213,"threads_started":1}
I20260926 20:53:03.874143 17277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:03.995479 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 17147
I20260926 20:53:04.017094 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 16765
I20260926 20:53:04.018276 16837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:04.133833 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 16765
2026-09-26T20:53:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7302 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:53:04 dist-test-slave-2dg3 krb5kdc[17435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:53:04 dist-test-slave-2dg3 krb5kdc[17435](info): set up 2 sockets
Sep 26 20:53:04 dist-test-slave-2dg3 krb5kdc[17435](info): commencing operation
krb5kdc: starting...
W20260926 20:53:05.235872 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:53:05 dist-test-slave-2dg3 krb5kdc[17435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:53:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:53:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:53:05.462235 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:42771
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:40125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:42771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:05.792229 17448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:05.792618 17448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:05.792794 17448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:05.800201 17448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:05.800372 17448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:05.800485 17448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:05.800596 17448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:05.800684 17448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:05.810750 17448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:40125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:42771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:42771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:05.812179 17448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:05.813582 17448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:05.826188 17456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:05.826151 17453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:05.826224 17454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:05.826548 17448 server_base.cc:1034] running on GCE node
I20260926 20:53:05.827821 17448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:05.829272 17448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:05.830513 17448 hybrid_clock.cc:648] HybridClock initialized: now 1790455985830463 us; error 42 us; skew 500 ppm
I20260926 20:53:05.830937 17448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T20:53:05Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1766 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:53:06 dist-test-slave-2dg3 krb5kdc[17462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:53:06 dist-test-slave-2dg3 krb5kdc[17462](info): set up 2 sockets
Sep 26 20:53:06 dist-test-slave-2dg3 krb5kdc[17462](info): commencing operation
krb5kdc: starting...
W20260926 20:53:06.999763 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:53:07 dist-test-slave-2dg3 krb5kdc[17462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:53:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:53:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:53:07.229760 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:46405
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:46405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:07.549885 17475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:07.550160 17475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:07.550354 17475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:07.557642 17475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:07.557750 17475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:07.557842 17475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:07.557945 17475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:07.558032 17475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:07.567765 17475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:46405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:46405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:07.568988 17475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:07.570226 17475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:07.581841 17481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:07.581876 17483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:07.581923 17480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:07.584080 17475 server_base.cc:1034] running on GCE node
I20260926 20:53:07.584914 17475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:07.586306 17475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:07.587502 17475 hybrid_clock.cc:648] HybridClock initialized: now 1790455987587464 us; error 37 us; skew 500 ppm
I20260926 20:53:07.587863 17475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:53:07 dist-test-slave-2dg3 krb5kdc[17462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455987, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:07.595501 17475 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:53:07.597133 17475 webserver.cc:460] Webserver started at http://127.8.249.190:46027/ using document root <none> and password file <none>
I20260926 20:53:07.597882 17475 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:07.598026 17475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:07.598330 17475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:07.601312 17475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "4c2ec68de6104f12ae8946164fb78fc3"
format_stamp: "Formatted at 2026-09-26 20:53:07 on dist-test-slave-2dg3"
server_key: "0db8cfc9921c0f83549947c1b37b8611"
server_key_iv: "e49b545b72bc680daf958adabb382e05"
server_key_version: "encryptionkey@0"
I20260926 20:53:07.602067 17475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "4c2ec68de6104f12ae8946164fb78fc3"
format_stamp: "Formatted at 2026-09-26 20:53:07 on dist-test-slave-2dg3"
server_key: "0db8cfc9921c0f83549947c1b37b8611"
server_key_iv: "e49b545b72bc680daf958adabb382e05"
server_key_version: "encryptionkey@0"
I20260926 20:53:07.607110 17475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 20:53:07.610627 17490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:07.612051 17475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:53:07.612236 17475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "4c2ec68de6104f12ae8946164fb78fc3"
format_stamp: "Formatted at 2026-09-26 20:53:07 on dist-test-slave-2dg3"
server_key: "0db8cfc9921c0f83549947c1b37b8611"
server_key_iv: "e49b545b72bc680daf958adabb382e05"
server_key_version: "encryptionkey@0"
I20260926 20:53:07.612437 17475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:07.640173 17475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:07.644339 17475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:07.644654 17475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:07.663069 17475 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:46405
I20260926 20:53:07.663079 17541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:46405 every 8 connection(s)
I20260926 20:53:07.664508 17475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:07.668242 17542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:07.671525 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 17475
I20260926 20:53:07.671743 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:53:07.672372 9190 external_mini_cluster.cc:1428] Setting key 2792e5e3b83625a97eb36deb9951ac3b
I20260926 20:53:07.681505 17542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3: Bootstrap starting.
I20260926 20:53:07.684732 17542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:07.685787 17542 log.cc:824] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:53:07 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:53:07.688993 17542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3: No bootstrap required, opened a new log
I20260926 20:53:07.695732 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:07.674375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.8.249.190:46405)
0926 20:53:07.675195 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:07.675207 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:07.675249 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:53:07.677393 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:07.677426 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:07.678044 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:07.678569 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:07.680331 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.681714 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.683047 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:07.683742 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:07.683941 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:07.688170 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:07.688178 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:07.688193 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:07.688228 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:07.691526 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:07.692382 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:07.692388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:07.692395 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:07.692577 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:07.692997 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:07.693000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:07.693002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:07.693525 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:07.693778 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:07.694203 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 20:53:07.694615 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":396,"threads_started":1}
I20260926 20:53:07.697808 17542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2ec68de6104f12ae8946164fb78fc3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46405 } }
I20260926 20:53:07.698256 17542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:07.698364 17542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2ec68de6104f12ae8946164fb78fc3, State: Initialized, Role: FOLLOWER
I20260926 20:53:07.698972 17542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2ec68de6104f12ae8946164fb78fc3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46405 } }
I20260926 20:53:07.699162 17542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:53:07.699250 17542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:53:07.699359 17542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:07.701340 17542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2ec68de6104f12ae8946164fb78fc3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46405 } }
I20260926 20:53:07.701766 17542 leader_election.cc:302] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c2ec68de6104f12ae8946164fb78fc3; no voters:
I20260926 20:53:07.702301 17542 leader_election.cc:288] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:53:07.702448 17547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:07.703720 17547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 1 LEADER]: Becoming Leader. State: Replica: 4c2ec68de6104f12ae8946164fb78fc3, State: Running, Role: LEADER
I20260926 20:53:07.704263 17547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2ec68de6104f12ae8946164fb78fc3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46405 } }
I20260926 20:53:07.704721 17542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:07.706934 17549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c2ec68de6104f12ae8946164fb78fc3. Latest consensus state: current_term: 1 leader_uuid: "4c2ec68de6104f12ae8946164fb78fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2ec68de6104f12ae8946164fb78fc3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46405 } } }
I20260926 20:53:07.706924 17548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c2ec68de6104f12ae8946164fb78fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2ec68de6104f12ae8946164fb78fc3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46405 } } }
I20260926 20:53:07.707367 17548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:07.708410 17549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:07.710131 17552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:07.715401 17552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:53:07.724527 17552 catalog_manager.cc:1347] Generated new cluster ID: b6e96a484a0d4bf481c64b829d436e23
I20260926 20:53:07.724742 17552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:53:07.736539 17552 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:53:07.737864 17552 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:53:07.744611 17552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3: Generated new TSK 0
I20260926 20:53:07.745467 17552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:53:07.802186 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46405
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:08.128866 17568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:08.129179 17568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:08.129288 17568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:08.136569 17568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:08.136694 17568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:08.136807 17568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:53:08.147385 17568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:08.148849 17568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:08.150224 17568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:08.163481 17576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:08.163565 17573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:08.163537 17574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:08.164382 17568 server_base.cc:1034] running on GCE node
I20260926 20:53:08.164968 17568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:08.165889 17568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:08.167076 17568 hybrid_clock.cc:648] HybridClock initialized: now 1790455988167016 us; error 50 us; skew 500 ppm
I20260926 20:53:08.167395 17568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:53:08 dist-test-slave-2dg3 krb5kdc[17462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455988, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:08.170406 17568 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:53:08.171811 17568 webserver.cc:460] Webserver started at http://127.8.249.129:36391/ using document root <none> and password file <none>
I20260926 20:53:08.172435 17568 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:08.172530 17568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:08.172807 17568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:08.175623 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "b7358f4899fe42d69d404ac4babd1a7f"
format_stamp: "Formatted at 2026-09-26 20:53:08 on dist-test-slave-2dg3"
server_key: "207906df1298682f4960f0793fb016bd"
server_key_iv: "fcdd1d46e50111cc3e817f015295bad8"
server_key_version: "encryptionkey@0"
I20260926 20:53:08.176326 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "b7358f4899fe42d69d404ac4babd1a7f"
format_stamp: "Formatted at 2026-09-26 20:53:08 on dist-test-slave-2dg3"
server_key: "207906df1298682f4960f0793fb016bd"
server_key_iv: "fcdd1d46e50111cc3e817f015295bad8"
server_key_version: "encryptionkey@0"
I20260926 20:53:08.181274 17568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:53:08.184624 17583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:08.185945 17568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:53:08.186120 17568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "b7358f4899fe42d69d404ac4babd1a7f"
format_stamp: "Formatted at 2026-09-26 20:53:08 on dist-test-slave-2dg3"
server_key: "207906df1298682f4960f0793fb016bd"
server_key_iv: "fcdd1d46e50111cc3e817f015295bad8"
server_key_version: "encryptionkey@0"
I20260926 20:53:08.186307 17568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:08.221047 17568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:08.225214 17568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:08.225510 17568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:08.226418 17568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:08.227736 17568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:08.227825 17568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:08.227999 17568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:08.228050 17568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:08.261025 17568 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:34527
I20260926 20:53:08.261112 17695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:34527 every 8 connection(s)
I20260926 20:53:08.262311 17568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:53:08.264724 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 17568
I20260926 20:53:08.264947 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:53:08.265651 9190 external_mini_cluster.cc:1428] Setting key 0a532cf538b24205634ada53159a3c97
Sep 26 20:53:08 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455988, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:53:08.283807 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:08.264966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:36535 (local address 127.8.249.190:46405)
0926 20:53:08.265443 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:08.265449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:08.266580 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 20:53:08.268580 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:08.268589 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:08.268703 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:08.268823 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:08.270235 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.271397 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.272866 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.273554 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.273704 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:08.276999 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:08.277009 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:08.277016 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:08.277059 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:08.280585 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:08.281277 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:08.281285 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:08.281291 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:08.281435 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:08.281823 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:08.281831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:08.281837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:08.282215 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:08.282379 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:08.283065 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 20:53:08.283326 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":226,"threads_started":1}
I20260926 20:53:08.284441 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:08.265591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:36535)
0926 20:53:08.266417 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:08.266470 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:08.268316 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:08.269013 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:08.269020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:08.269522 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:08.270076 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:08.270093 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.271523 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.271527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:08.272718 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:08.272729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.273768 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.273773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:08.273957 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:08.274671 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:08.274696 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:08.276790 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:08.280728 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:08.280735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:08.280749 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:08.281158 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:08.281540 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:08.281543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:08.281545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:08.281710 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:08.282332 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:08.282337 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:08.282648 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 20:53:08.282942 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 20:53:08.283330 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":355,"threads_started":1}
I20260926 20:53:08.286230 17696 heartbeater.cc:347] Connected to a master server at 127.8.249.190:46405
I20260926 20:53:08.286564 17696 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:08.287531 17696 heartbeater.cc:511] Master 127.8.249.190:46405 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
I20260926 20:53:08.289933 17507 ts_manager.cc:194] Registered new tserver with Master: b7358f4899fe42d69d404ac4babd1a7f (127.8.249.129:34527)
I20260926 20:53:08.291548 17507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:36535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:53:08.318292 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46405
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:08.645715 17702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:08.646000 17702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:08.646090 17702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:08.653483 17702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:08.653616 17702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:08.653734 17702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:53:08.663609 17702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:08.664914 17702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:08.666257 17702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:08.679611 17708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:08.679814 17710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:08.680781 17702 server_base.cc:1034] running on GCE node
W20260926 20:53:08.681336 17707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:08.682185 17702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:08.683369 17702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:08.684614 17702 hybrid_clock.cc:648] HybridClock initialized: now 1790455988684548 us; error 53 us; skew 500 ppm
I20260926 20:53:08.685073 17702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:53:08 dist-test-slave-2dg3 krb5kdc[17462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455988, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:08.688350 17702 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:53:08.689960 17702 webserver.cc:460] Webserver started at http://127.8.249.130:43509/ using document root <none> and password file <none>
I20260926 20:53:08.690737 17702 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:08.690901 17702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:08.691229 17702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:08.694168 17702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "7e68f7f1565849ecb28a1435ae6f2885"
format_stamp: "Formatted at 2026-09-26 20:53:08 on dist-test-slave-2dg3"
server_key: "812c7d0c6214cb49730ffd84c4109997"
server_key_iv: "2ee8d7ab2d7ec22301a3f09d017f2d85"
server_key_version: "encryptionkey@0"
I20260926 20:53:08.695084 17702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "7e68f7f1565849ecb28a1435ae6f2885"
format_stamp: "Formatted at 2026-09-26 20:53:08 on dist-test-slave-2dg3"
server_key: "812c7d0c6214cb49730ffd84c4109997"
server_key_iv: "2ee8d7ab2d7ec22301a3f09d017f2d85"
server_key_version: "encryptionkey@0"
I20260926 20:53:08.700294 17702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 20:53:08.703734 17717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:08.705070 17702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:53:08.705252 17702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "7e68f7f1565849ecb28a1435ae6f2885"
format_stamp: "Formatted at 2026-09-26 20:53:08 on dist-test-slave-2dg3"
server_key: "812c7d0c6214cb49730ffd84c4109997"
server_key_iv: "2ee8d7ab2d7ec22301a3f09d017f2d85"
server_key_version: "encryptionkey@0"
I20260926 20:53:08.705446 17702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:08.718283 17702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:08.722676 17702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:08.723068 17702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:08.724082 17702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:08.725484 17702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:08.725597 17702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:08.725751 17702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:08.725824 17702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:08.759532 17702 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:35849
I20260926 20:53:08.759583 17829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:35849 every 8 connection(s)
I20260926 20:53:08.760864 17702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:53:08.770699 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 17702
I20260926 20:53:08.770910 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:53:08.771486 9190 external_mini_cluster.cc:1428] Setting key ab065726483ee1635925d7aeee3ab3bd
Sep 26 20:53:08 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455988, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:53:08.781776 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:08.763490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:60105 (local address 127.8.249.190:46405)
0926 20:53:08.763697 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:08.763703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:08.764986 (+ 1283us) server_negotiation.cc:407] Connection header received
0926 20:53:08.766763 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:08.766767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:08.766907 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:08.766996 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:08.768488 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.769641 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.771754 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.772186 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.772296 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:08.775990 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:08.775995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:08.775999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:08.776033 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:08.778683 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:08.779468 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:08.779476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:08.779479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:08.779617 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:08.780055 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:08.780060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:08.780063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:08.780335 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:08.780489 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:08.781205 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 20:53:08.781439 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:53:08.782869 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:08.764121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:60105)
0926 20:53:08.764825 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:08.764863 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:08.766490 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:08.767233 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:08.767241 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:08.767766 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:08.768346 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:08.768361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.769788 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.769792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:08.771588 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:08.771612 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:08.772429 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:08.772438 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:08.772664 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:08.773385 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:08.773420 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:08.775741 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:08.778862 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:08.778872 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:08.778891 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:08.779336 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:08.779729 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:08.779734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:08.779738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:08.779932 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:08.780461 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:08.780469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:08.780831 (+ 362us) client_negotiation.cc:769] Sending connection context
0926 20:53:08.781100 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 20:53:08.781672 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":269,"threads_started":1}
I20260926 20:53:08.784632 17830 heartbeater.cc:347] Connected to a master server at 127.8.249.190:46405
I20260926 20:53:08.784978 17830 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:08.785717 17830 heartbeater.cc:511] Master 127.8.249.190:46405 requested a full tablet report, sending...
I20260926 20:53:08.787279 17507 ts_manager.cc:194] Registered new tserver with Master: 7e68f7f1565849ecb28a1435ae6f2885 (127.8.249.130:35849)
I20260926 20:53:08.788599 17507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:60105
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:53:08.822745 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46405
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:09.169261 17835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:09.169562 17835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:09.169737 17835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:09.177202 17835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:09.177320 17835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:09.177523 17835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:53:09.187654 17835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:09.188977 17835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:09.190368 17835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:09.203814 17840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:09.203781 17843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:09.203722 17841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:09.204695 17835 server_base.cc:1034] running on GCE node
I20260926 20:53:09.205375 17835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:09.206377 17835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:09.207592 17835 hybrid_clock.cc:648] HybridClock initialized: now 1790455989207542 us; error 46 us; skew 500 ppm
I20260926 20:53:09.207922 17835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:53:09 dist-test-slave-2dg3 krb5kdc[17462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:09.211688 17835 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:53:09.213680 17835 webserver.cc:460] Webserver started at http://127.8.249.131:45137/ using document root <none> and password file <none>
I20260926 20:53:09.214655 17835 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:09.214821 17835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:09.215190 17835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:09.219352 17835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "c05fe21c692741c98e6711c0ecf6f32a"
format_stamp: "Formatted at 2026-09-26 20:53:09 on dist-test-slave-2dg3"
server_key: "f90d7db423e00b098e58f760de0dd8d2"
server_key_iv: "e9c01e85bdc8d06d9ff86d12199b9b44"
server_key_version: "encryptionkey@0"
I20260926 20:53:09.220320 17835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "c05fe21c692741c98e6711c0ecf6f32a"
format_stamp: "Formatted at 2026-09-26 20:53:09 on dist-test-slave-2dg3"
server_key: "f90d7db423e00b098e58f760de0dd8d2"
server_key_iv: "e9c01e85bdc8d06d9ff86d12199b9b44"
server_key_version: "encryptionkey@0"
I20260926 20:53:09.225571 17835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:53:09.229080 17850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:09.230427 17835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:53:09.230602 17835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "c05fe21c692741c98e6711c0ecf6f32a"
format_stamp: "Formatted at 2026-09-26 20:53:09 on dist-test-slave-2dg3"
server_key: "f90d7db423e00b098e58f760de0dd8d2"
server_key_iv: "e9c01e85bdc8d06d9ff86d12199b9b44"
server_key_version: "encryptionkey@0"
I20260926 20:53:09.230826 17835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:09.254372 17835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:09.258940 17835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:09.259239 17835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:09.260203 17835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:09.261656 17835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:09.261749 17835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:09.261919 17835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:09.261967 17835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:09.294862 17835 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:39685
I20260926 20:53:09.294955 17962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:39685 every 8 connection(s)
I20260926 20:53:09.295285 17696 heartbeater.cc:503] Master 127.8.249.190:46405 was elected leader, sending a full tablet report...
I20260926 20:53:09.296315 17835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:53:09.304854 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 17835
I20260926 20:53:09.305032 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:53:09.305452 9190 external_mini_cluster.cc:1428] Setting key d327579e09ca2123a472dd4af427f2f8
Sep 26 20:53:09 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:53:09.316450 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:09.298849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:43253 (local address 127.8.249.190:46405)
0926 20:53:09.299273 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:09.299279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:09.300273 (+ 994us) server_negotiation.cc:407] Connection header received
0926 20:53:09.301954 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:09.301958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:09.302028 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:09.302142 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:09.303642 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:09.304469 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:09.306301 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:09.306919 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:09.307052 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:09.310686 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:09.310691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:09.310694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:09.310725 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:09.313533 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:09.314241 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:09.314246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:09.314248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:09.314373 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:09.314737 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:09.314740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:09.314742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:09.315088 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:09.315211 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:09.315910 (+ 699us) server_negotiation.cc:299] Negotiation successful
0926 20:53:09.316091 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":201,"threads_started":1}
I20260926 20:53:09.317412 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:09.299456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:43253)
0926 20:53:09.300119 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:09.300164 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:09.301692 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:09.302333 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:09.302340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:09.302932 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:09.303478 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:09.303493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:09.304633 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:09.304636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:09.306159 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:09.306174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:09.307110 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:09.307115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:09.307403 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:09.308128 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:09.308158 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:09.310499 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:09.313677 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:09.313684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:09.313698 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:09.314131 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:09.314470 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:09.314474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:09.314476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:09.314643 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:09.315199 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:09.315205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:09.315532 (+ 327us) client_negotiation.cc:769] Sending connection context
0926 20:53:09.315776 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 20:53:09.316165 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":252,"threads_started":1}
I20260926 20:53:09.318943 17963 heartbeater.cc:347] Connected to a master server at 127.8.249.190:46405
I20260926 20:53:09.319284 17963 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:09.320142 17963 heartbeater.cc:511] Master 127.8.249.190:46405 requested a full tablet report, sending...
I20260926 20:53:09.321837 17507 ts_manager.cc:194] Registered new tserver with Master: c05fe21c692741c98e6711c0ecf6f32a (127.8.249.131:39685)
I20260926 20:53:09.322662 17507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:43253
I20260926 20:53:09.333844 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab.
Entry for principal oryx/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab.
WARNING: no policy specified for oryx/127.8.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab.
Entry for principal oryx/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab.
WARNING: no policy specified for oryx/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab.
Entry for principal oryx/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab.
I20260926 20:53:09.467912 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 17568
Sep 26 20:53:09 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:53:09.483624 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:09.468969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.8.249.129:34527)
0926 20:53:09.469552 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:09.469559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:09.469587 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:53:09.469698 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:09.469705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:09.469934 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:09.470064 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:09.471394 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:09.472369 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:09.473532 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:09.474072 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:09.474164 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:09.478206 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:09.478216 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:09.478230 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:09.478266 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:09.480875 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:09.481606 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:09.481613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:09.481615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:09.481735 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:09.482076 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:09.482079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:09.482081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:09.482427 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:09.482646 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:09.483021 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 20:53:09.483218 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":296,"threads_started":1}
I20260926 20:53:09.484901 17691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:09.567281 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 17568
I20260926 20:53:09.583900 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 17702
Sep 26 20:53:09 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:53:09.599367 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:09.584882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.8.249.130:35849)
0926 20:53:09.585341 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:09.585347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:09.585372 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:53:09.585430 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:09.585436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:09.585652 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:09.585771 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:09.587002 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:09.587984 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:09.589164 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:09.589680 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:09.589779 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:09.593981 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:09.593993 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:09.594008 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:09.594042 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:09.596835 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:09.597460 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:09.597467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:09.597469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:09.597584 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:09.597881 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:09.597884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:09.597886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:09.598208 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:09.598425 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:09.598724 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 20:53:09.598976 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":216,"threads_started":1}
I20260926 20:53:09.600659 17825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:09.683261 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 17702
I20260926 20:53:09.700490 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 17835
Sep 26 20:53:09 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:53:09.717773 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:09.701482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.8.249.131:39685)
0926 20:53:09.702111 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:09.702119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:09.702146 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:53:09.702205 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:09.702211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:09.702423 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:09.702599 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:09.703951 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:09.705080 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:09.706508 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:09.707165 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:09.707323 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:09.711883 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:09.711897 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:09.711912 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:09.711956 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:09.714649 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:09.715490 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:09.715499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:09.715501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:09.715643 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:09.716053 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:09.716058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:09.716060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:09.716466 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:09.716823 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:09.717116 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 20:53:09.717404 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":315,"threads_started":1}
I20260926 20:53:09.719156 17958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:09.802419 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 17835
I20260926 20:53:09.819000 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 17475
I20260926 20:53:09.820165 17537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:09.921325 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 17475
I20260926 20:53:09.941711 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:46405
--webserver_interface=127.8.249.190
--webserver_port=46027
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:46405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:10.260721 17981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:10.261075 17981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:10.261231 17981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:10.268401 17981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:10.268543 17981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:10.268646 17981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:10.268744 17981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:10.268823 17981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:10.278589 17981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:46405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:46405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=46027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:10.279965 17981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:10.281273 17981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:10.293322 17986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:10.293394 17987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:10.293413 17989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:10.294130 17981 server_base.cc:1034] running on GCE node
I20260926 20:53:10.295004 17981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:10.296435 17981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:10.297672 17981 hybrid_clock.cc:648] HybridClock initialized: now 1790455990297636 us; error 48 us; skew 500 ppm
I20260926 20:53:10.298014 17981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:53:10 dist-test-slave-2dg3 krb5kdc[17462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:10.301077 17981 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:53:10.302501 17981 webserver.cc:460] Webserver started at http://127.8.249.190:46027/ using document root <none> and password file <none>
I20260926 20:53:10.303234 17981 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:10.303385 17981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:10.308672 17981 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 20:53:10.311491 17996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:10.312825 17981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:53:10.313014 17981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "4c2ec68de6104f12ae8946164fb78fc3"
format_stamp: "Formatted at 2026-09-26 20:53:07 on dist-test-slave-2dg3"
server_key: "0db8cfc9921c0f83549947c1b37b8611"
server_key_iv: "e49b545b72bc680daf958adabb382e05"
server_key_version: "encryptionkey@0"
I20260926 20:53:10.314002 17981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:10.335326 17981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:10.339746 17981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:10.340004 17981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:10.357932 17981 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:46405
I20260926 20:53:10.357954 18047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:46405 every 8 connection(s)
I20260926 20:53:10.359547 17981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:10.363070 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 17981
I20260926 20:53:10.363858 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:34527
--local_ip_for_outbound_sockets=127.8.249.129
--tserver_master_addrs=127.8.249.190:46405
--webserver_port=36391
--webserver_interface=127.8.249.129
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.249.129
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:53:10.366385 18048 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:53:10.369557 18048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3: Bootstrap starting.
I20260926 20:53:10.396623 18048 log.cc:824] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:10.407137 18048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:53:10.407704 18048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3: Bootstrap complete.
I20260926 20:53:10.421103 18048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2ec68de6104f12ae8946164fb78fc3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46405 } }
I20260926 20:53:10.422421 18048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2ec68de6104f12ae8946164fb78fc3, State: Initialized, Role: FOLLOWER
I20260926 20:53:10.423192 18048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2ec68de6104f12ae8946164fb78fc3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46405 } }
I20260926 20:53:10.423472 18048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:53:10.423592 18048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:53:10.423774 18048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:53:10.430011 18048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2ec68de6104f12ae8946164fb78fc3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46405 } }
I20260926 20:53:10.430846 18048 leader_election.cc:302] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c2ec68de6104f12ae8946164fb78fc3; no voters:
I20260926 20:53:10.431586 18048 leader_election.cc:288] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:53:10.431797 18052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:53:10.433743 18052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [term 2 LEADER]: Becoming Leader. State: Replica: 4c2ec68de6104f12ae8946164fb78fc3, State: Running, Role: LEADER
I20260926 20:53:10.434024 18048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:10.434363 18052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2ec68de6104f12ae8946164fb78fc3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46405 } }
I20260926 20:53:10.440368 18053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4c2ec68de6104f12ae8946164fb78fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2ec68de6104f12ae8946164fb78fc3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46405 } } }
I20260926 20:53:10.440829 18053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:10.441464 18054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c2ec68de6104f12ae8946164fb78fc3. Latest consensus state: current_term: 2 leader_uuid: "4c2ec68de6104f12ae8946164fb78fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2ec68de6104f12ae8946164fb78fc3" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46405 } } }
I20260926 20:53:10.441900 18054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:10.452299 18061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:10.458748 18061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:53:10.460283 18061 catalog_manager.cc:1259] Loaded cluster ID: b6e96a484a0d4bf481c64b829d436e23
I20260926 20:53:10.460402 18061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:53:10.464808 18061 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:53:10.465467 18069 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:53:10.466528 18061 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4c2ec68de6104f12ae8946164fb78fc3: Loaded TSK: 0
I20260926 20:53:10.468181 18061 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:53:10.728171 18050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:10.728492 18050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:10.728655 18050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:10.736140 18050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:10.736301 18050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:10.736476 18050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:53:10.746773 18050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.129.keytab
--principal=oryx/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:34527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=36391
--tserver_master_addrs=127.8.249.190:46405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:10.748391 18050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:10.749706 18050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:10.765957 18074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:10.765977 18075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:10.766564 18077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:10.766821 18050 server_base.cc:1034] running on GCE node
I20260926 20:53:10.767529 18050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:10.768625 18050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:10.769840 18050 hybrid_clock.cc:648] HybridClock initialized: now 1790455990769802 us; error 36 us; skew 500 ppm
I20260926 20:53:10.770164 18050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:53:10 dist-test-slave-2dg3 krb5kdc[17462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455990, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:10.773693 18050 init.cc:377] Logged in from keytab as oryx/127.8.249.129@KRBTEST.COM (short username oryx)
I20260926 20:53:10.775342 18050 webserver.cc:460] Webserver started at http://127.8.249.129:36391/ using document root <none> and password file <none>
I20260926 20:53:10.776048 18050 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:10.776199 18050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:10.781905 18050 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:53:10.784888 18084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:10.786319 18050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:53:10.786489 18050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "b7358f4899fe42d69d404ac4babd1a7f"
format_stamp: "Formatted at 2026-09-26 20:53:08 on dist-test-slave-2dg3"
server_key: "207906df1298682f4960f0793fb016bd"
server_key_iv: "fcdd1d46e50111cc3e817f015295bad8"
server_key_version: "encryptionkey@0"
I20260926 20:53:10.787508 18050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:10.811468 18050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:10.816057 18050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:10.816344 18050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:10.817436 18050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:10.819037 18050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:10.819164 18050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:10.819304 18050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:10.819379 18050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:10.859740 18050 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:34527
I20260926 20:53:10.859829 18196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:34527 every 8 connection(s)
I20260926 20:53:10.861261 18050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:53:10.870556 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 18050
I20260926 20:53:10.871225 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:35849
--local_ip_for_outbound_sockets=127.8.249.130
--tserver_master_addrs=127.8.249.190:46405
--webserver_port=43509
--webserver_interface=127.8.249.130
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.249.130
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:53:10 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:10 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:53:11.200405 18201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:11.200685 18201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:11.200829 18201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:11.208261 18201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:11.208384 18201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:11.208595 18201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:53:11.222038 18201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.130.keytab
--principal=oryx/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:35849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=43509
--tserver_master_addrs=127.8.249.190:46405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:11.223551 18201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:11.224936 18201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:11.239228 18208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:11.239200 18207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:11.243105 18210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:11.247189 18201 server_base.cc:1034] running on GCE node
I20260926 20:53:11.247915 18201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:11.248986 18201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:11.250208 18201 hybrid_clock.cc:648] HybridClock initialized: now 1790455991250167 us; error 35 us; skew 500 ppm
I20260926 20:53:11.250540 18201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:53:11 dist-test-slave-2dg3 krb5kdc[17462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455991, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:11.254382 18201 init.cc:377] Logged in from keytab as oryx/127.8.249.130@KRBTEST.COM (short username oryx)
I20260926 20:53:11.256168 18201 webserver.cc:460] Webserver started at http://127.8.249.130:43509/ using document root <none> and password file <none>
I20260926 20:53:11.256970 18201 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:11.257143 18201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:11.262581 18201 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:53:11.265671 18217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:11.267499 18201 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 20:53:11.267668 18201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "7e68f7f1565849ecb28a1435ae6f2885"
format_stamp: "Formatted at 2026-09-26 20:53:08 on dist-test-slave-2dg3"
server_key: "812c7d0c6214cb49730ffd84c4109997"
server_key_iv: "2ee8d7ab2d7ec22301a3f09d017f2d85"
server_key_version: "encryptionkey@0"
I20260926 20:53:11.268615 18201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:11.304281 18201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:11.309012 18201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:11.309302 18201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:11.310374 18201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:11.311854 18201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:11.311970 18201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:11.312099 18201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:11.312184 18201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:11.364151 18201 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:35849
I20260926 20:53:11.364346 18329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:35849 every 8 connection(s)
I20260926 20:53:11.365931 18201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:53:11.366866 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 18201
I20260926 20:53:11.367549 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:39685
--local_ip_for_outbound_sockets=127.8.249.131
--tserver_master_addrs=127.8.249.190:46405
--webserver_port=45137
--webserver_interface=127.8.249.131
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.249.131
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:53:11 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:11 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:53:11.735241 18333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:11.735610 18333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:11.735772 18333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:11.743251 18333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:11.743417 18333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:11.743585 18333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:53:11.753988 18333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/krb5kdc/oryx_127.8.249.131.keytab
--principal=oryx/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:39685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=45137
--tserver_master_addrs=127.8.249.190:46405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:11.755475 18333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:11.756856 18333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:11.775131 18341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:11.775126 18340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:11.784943 18333 server_base.cc:1034] running on GCE node
W20260926 20:53:11.785650 18343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:11.786460 18333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:11.787650 18333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:11.788872 18333 hybrid_clock.cc:648] HybridClock initialized: now 1790455991788806 us; error 57 us; skew 500 ppm
I20260926 20:53:11.789203 18333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:53:11 dist-test-slave-2dg3 krb5kdc[17462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790455991, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:11.792598 18333 init.cc:377] Logged in from keytab as oryx/127.8.249.131@KRBTEST.COM (short username oryx)
I20260926 20:53:11.794399 18333 webserver.cc:460] Webserver started at http://127.8.249.131:45137/ using document root <none> and password file <none>
I20260926 20:53:11.795169 18333 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:11.795329 18333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:11.800724 18333 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 20:53:11.803658 18350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:11.805083 18333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:53:11.805260 18333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "c05fe21c692741c98e6711c0ecf6f32a"
format_stamp: "Formatted at 2026-09-26 20:53:09 on dist-test-slave-2dg3"
server_key: "f90d7db423e00b098e58f760de0dd8d2"
server_key_iv: "e9c01e85bdc8d06d9ff86d12199b9b44"
server_key_version: "encryptionkey@0"
I20260926 20:53:11.806166 18333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:11.840786 18333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:11.845417 18333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:11.845685 18333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:11.846827 18333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:11.848380 18333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:11.848493 18333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:11.848618 18333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:11.848701 18333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:11.916270 18333 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:39685
I20260926 20:53:11.916364 18462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:39685 every 8 connection(s)
I20260926 20:53:11.919114 18333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:53:11.929090 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 18333
Sep 26 20:53:11 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:11 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:12.005679 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:11.961606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.8.249.190:46405)
0926 20:53:11.967437 (+ 5831us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:11.967447 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:11.967469 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:53:11.967549 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:11.967558 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:11.967671 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:11.974157 (+ 6486us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:11.974469 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:11.976424 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:11.980782 (+ 4358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:11.981850 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:11.982029 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:11.987596 (+ 5567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:11.987612 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:11.987649 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:11.987738 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:11.994202 (+ 6464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:11.995548 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:11.995564 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:11.995580 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:11.995928 (+ 348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:11.999536 (+ 3608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:11.999546 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:11.999553 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:12.000543 (+ 990us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:12.001036 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:12.003328 (+ 2292us) server_negotiation.cc:299] Negotiation successful
0926 20:53:12.004089 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5631,"thread_start_us":6270,"threads_started":1}
W20260926 20:53:12.217530 18200 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.341s user 0.000s sys 0.014s
W20260926 20:53:12.219010 18199 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:10.864484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:46199 (local address 127.8.249.190:46405)
0926 20:53:10.865277 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:10.865289 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:10.865578 (+ 289us) server_negotiation.cc:407] Connection header received
0926 20:53:10.867878 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:10.867893 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:10.868290 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:10.868612 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:10.870067 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.871672 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.874156 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.874927 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.875087 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:12.218641 (+1343554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:46199: BlockingRecv error: recv got EOF from 127.8.249.129:46199 (error 108)
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":348,"threads_started":1}
I20260926 20:53:12.219532 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:10.864563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:46199)
0926 20:53:10.865393 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:10.865438 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:10.867233 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:10.868757 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:10.868764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:10.869338 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:10.869853 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.869869 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.871842 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.871851 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.873979 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:10.873998 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:10.875172 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:10.875181 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:10.875491 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:10.876371 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:10.876405 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:12.218383 (+1341978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":349,"threads_started":1}
W20260926 20:53:12.220335 18197 heartbeater.cc:644] Failed to heartbeat to 127.8.249.190:46405 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.249.190:46405: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Sep 26 20:53:12 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:12 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:12.230422 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:12.221213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:33243)
0926 20:53:12.221403 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:12.221422 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:12.221535 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:12.221897 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:12.221902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:12.222266 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:12.222418 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:12.222425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.223623 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.223629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:12.225043 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:12.225056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.225803 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.225809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:12.225972 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:12.226599 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:12.226616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:12.230127 (+ 3511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 20:53:12.230573 18199 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:12.221471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:33243 (local address 127.8.249.190:46405)
0926 20:53:12.221634 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:12.221639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:12.221655 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:53:12.221710 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:12.221713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:12.221769 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:12.221869 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:12.222558 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.223477 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.225230 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.225623 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.225762 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:12.230327 (+ 4565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:33243: BlockingRecv error: recv got EOF from 127.8.249.129:33243 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 20:53:12 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:12 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:12.240706 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:12.231383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:56863)
0926 20:53:12.231516 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:12.231535 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:12.231634 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:12.232082 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:12.232087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:12.232397 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:12.232538 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:12.232546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.233509 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.233514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:12.234610 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:12.234621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.235719 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.235724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:12.235915 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:12.236546 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:12.236568 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:12.240429 (+ 3861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 20:53:12.240849 18199 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:12.231675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:56863 (local address 127.8.249.190:46405)
0926 20:53:12.231831 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:12.231835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:12.231847 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:53:12.231887 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:12.231890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:12.231950 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:12.232053 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:12.232670 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.233376 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.234745 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.235408 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.235527 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:12.240607 (+ 5080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:56863: BlockingRecv error: recv got EOF from 127.8.249.129:56863 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 20:53:12.241339 18197 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:53:12.835494 18332 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.437s user 0.012s sys 0.007s
W20260926 20:53:12.837087 18331 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:11.369930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:35603 (local address 127.8.249.190:46405)
0926 20:53:11.385789 (+ 15859us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:11.385800 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:11.385843 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:53:11.388611 (+ 2768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:11.388621 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:11.388786 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:11.388977 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:11.391430 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:11.393228 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:11.395960 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:11.396954 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:11.397136 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:12.836648 (+1439512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:35603: BlockingRecv error: recv got EOF from 127.8.249.130:35603 (error 108)
Metrics: {"server-negotiator.queue_time_us":15629,"thread_start_us":15526,"threads_started":1}
I20260926 20:53:12.837298 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:11.371023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:35603)
0926 20:53:11.380664 (+ 9641us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:11.380761 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:11.388233 (+ 7472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:11.389275 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:11.389292 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:11.390175 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:11.391176 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:11.391207 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:11.393441 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:11.393449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:11.395768 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:11.395785 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:11.397241 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:11.397251 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:11.397596 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:11.398575 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:11.398621 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:12.836234 (+1437613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":9094,"thread_start_us":11609,"threads_started":1}
W20260926 20:53:12.838505 18330 heartbeater.cc:644] Failed to heartbeat to 127.8.249.190:46405 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.249.190:46405: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Sep 26 20:53:12 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:12 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:12.857868 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:12.839421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:39059)
0926 20:53:12.839704 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:12.839734 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:12.839907 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:12.840273 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:12.840280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:12.840952 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:12.841264 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:12.841281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.842529 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.842539 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:12.844912 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:12.844934 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.845691 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.845701 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:12.845969 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:12.852234 (+ 6265us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:12.852272 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:12.857560 (+ 5288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 20:53:12.858006 18331 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:12.839676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:39059 (local address 127.8.249.190:46405)
0926 20:53:12.839839 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:12.839843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:12.839860 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:53:12.840046 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:12.840049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:12.840110 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:12.840189 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:12.841436 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.842323 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.845086 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.845472 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.845567 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:12.857776 (+ 12209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:39059: BlockingRecv error: recv got EOF from 127.8.249.130:39059 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 20:53:12 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:12 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:53:12.872375 18331 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:12.859355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:33459 (local address 127.8.249.190:46405)
0926 20:53:12.859496 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:12.859501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:12.859519 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:53:12.859704 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:12.859708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:12.859787 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:12.860053 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:12.861298 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.862300 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.864995 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.865387 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.865479 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:12.872142 (+ 6663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:33459: BlockingRecv error: recv got EOF from 127.8.249.130:33459 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:53:12.872444 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:12.859113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:33459)
0926 20:53:12.859378 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:12.859406 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:12.859556 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:12.860171 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:12.860179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:12.860809 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:12.861103 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:12.861121 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.862464 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.862472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:12.864822 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:12.864845 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:12.865565 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:12.865575 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:12.865917 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:12.866688 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:12.866723 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:12.872006 (+ 5283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 20:53:12.873037 18330 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:53:13.246752 18465 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.256s user 0.008s sys 0.019s
W20260926 20:53:13.248158 18466 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:11.925266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:44511 (local address 127.8.249.190:46405)
0926 20:53:11.961348 (+ 36082us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:11.961359 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:11.966875 (+ 5516us) server_negotiation.cc:407] Connection header received
0926 20:53:11.976723 (+ 9848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:11.976732 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:11.976842 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:11.976943 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:11.980739 (+ 3796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:11.982887 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:11.987116 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:11.988306 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:11.988483 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:13.247843 (+1259360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:44511: BlockingRecv error: recv got EOF from 127.8.249.131:44511 (error 108)
Metrics: {"server-negotiator.queue_time_us":35892,"thread_start_us":36097,"threads_started":1}
I20260926 20:53:13.248771 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:11.924371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:44511)
0926 20:53:11.964491 (+ 40120us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:11.964575 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:11.969766 (+ 5191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:11.977933 (+ 8167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:11.977953 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:11.979183 (+ 1230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:11.980427 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:11.980480 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:11.983225 (+ 2745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:11.983236 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:11.986774 (+ 3538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:11.986821 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:11.989136 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:11.989148 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:11.989673 (+ 525us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:11.990784 (+ 1111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:11.990909 (+ 125us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:13.247631 (+1256722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38208,"thread_start_us":38257,"threads_started":1}
W20260926 20:53:13.249794 18463 heartbeater.cc:644] Failed to heartbeat to 127.8.249.190:46405 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.249.190:46405: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Sep 26 20:53:13 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:13 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:13.255614 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:13.241862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:34191)
0926 20:53:13.245884 (+ 4022us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:13.245903 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:13.246067 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:13.246459 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:13.246464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:13.246983 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:13.247183 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:13.247201 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.248268 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.248274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:13.249558 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:13.249572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.250375 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.250382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:13.250566 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:13.251256 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:13.251283 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:13.255268 (+ 3985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3881,"thread_start_us":3729,"threads_started":1}
W20260926 20:53:13.255681 18331 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:13.241965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:34191 (local address 127.8.249.190:46405)
0926 20:53:13.242170 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:13.242178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:13.246018 (+ 3840us) server_negotiation.cc:407] Connection header received
0926 20:53:13.246211 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:13.246216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:13.246308 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:13.246423 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:13.247335 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.248104 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.249711 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.250174 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.250327 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:13.255422 (+ 5095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:34191: BlockingRecv error: recv got EOF from 127.8.249.129:34191 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 20:53:13 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:13 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:13.259164 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:13.250448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:53771)
0926 20:53:13.250633 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:13.250654 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:13.250780 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:13.251159 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:13.251166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:13.251609 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:13.251839 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:13.251851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.253122 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.253127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:13.254122 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:13.254133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.254983 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.254990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:13.255168 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:13.255902 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:13.255923 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:13.258968 (+ 3045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 20:53:13.259356 18466 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:13.250547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:53771 (local address 127.8.249.190:46405)
0926 20:53:13.250779 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:13.250783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:13.250842 (+ 59us) server_negotiation.cc:407] Connection header received
0926 20:53:13.250923 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:13.250929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:13.251000 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:13.251112 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:13.252011 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.252938 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.254257 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.254792 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.254944 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:13.259131 (+ 4187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:53771: BlockingRecv error: recv got EOF from 127.8.249.131:53771 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 26 20:53:13 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:13 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:13.267725 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:13.259842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:50911)
0926 20:53:13.260016 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:13.260035 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:13.260139 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:13.260483 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:13.260487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:13.260875 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:13.261018 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:13.261025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.262013 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.262018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:13.263034 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:13.263044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.263724 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.263730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:13.263919 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:13.264497 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:13.264513 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:13.267453 (+ 2940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 20:53:13.267882 18466 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:13.259903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:50911 (local address 127.8.249.190:46405)
0926 20:53:13.260049 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:13.260057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:13.260076 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:53:13.260271 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:13.260276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:13.260357 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:13.260454 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:13.261146 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.261872 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.263163 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.263570 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.263707 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:13.267618 (+ 3911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:50911: BlockingRecv error: recv got EOF from 127.8.249.131:50911 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 20:53:13.268070 18463 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 20:53:13 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:13 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:13.882081 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:13.873562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:43773)
0926 20:53:13.874073 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:13.874095 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:13.874208 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:13.874492 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:13.874495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:13.874911 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:13.875069 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:13.875079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.875993 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.875996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:13.876955 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:13.876966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.877596 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.877600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:13.877714 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:13.878252 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:13.878268 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:13.881845 (+ 3577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":285,"threads_started":1}
W20260926 20:53:13.882313 18472 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:13.873629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:43773 (local address 127.8.249.190:46405)
0926 20:53:13.874082 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:13.874086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:13.874169 (+ 83us) server_negotiation.cc:407] Connection header received
0926 20:53:13.874302 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:13.874306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:13.874363 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:13.874449 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:13.875185 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.875877 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.877076 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:13.877448 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:13.877534 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:13.882039 (+ 4505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:43773: BlockingRecv error: recv got EOF from 127.8.249.130:43773 (error 108)
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":257,"threads_started":1}
Sep 26 20:53:14 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:14 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:14.265178 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:14.256535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:52907)
0926 20:53:14.257013 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:14.257028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:14.257205 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:14.257535 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:14.257540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:14.257964 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:14.258144 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:14.258153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.259223 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.259229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:14.260123 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:14.260133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.260802 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.260809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:14.260976 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:14.261522 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:14.261539 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:14.264948 (+ 3409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":279,"threads_started":1}
W20260926 20:53:14.265350 18472 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:14.256579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:52907 (local address 127.8.249.190:46405)
0926 20:53:14.256786 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:14.256794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:14.257133 (+ 339us) server_negotiation.cc:407] Connection header received
0926 20:53:14.257324 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:14.257329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:14.257400 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:14.257501 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:14.258283 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.259095 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.260257 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.260652 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.260786 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:14.265125 (+ 4339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:52907: BlockingRecv error: recv got EOF from 127.8.249.129:52907 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 20:53:14 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:14 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:14.276618 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:14.268560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:44887)
0926 20:53:14.269097 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:14.269112 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:14.269214 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:14.269470 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:14.269473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:14.269833 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:14.269966 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:14.269974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.270936 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.270940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:14.271934 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:14.271945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.272674 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.272679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:14.272790 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:14.273268 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:14.273284 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:14.276393 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":306,"threads_started":1}
W20260926 20:53:14.276758 18472 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:14.268642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:44887 (local address 127.8.249.190:46405)
0926 20:53:14.268787 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:14.268791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:14.269176 (+ 385us) server_negotiation.cc:407] Connection header received
0926 20:53:14.269298 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:14.269301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:14.269355 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:14.269429 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:14.270072 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.270784 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.272044 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.272498 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.272591 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:14.276561 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:44887: BlockingRecv error: recv got EOF from 127.8.249.131:44887 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 20:53:14 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:14 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:14.892210 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:14.883074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:41781)
0926 20:53:14.883614 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:14.883642 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:14.883782 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:14.884085 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:14.884088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:14.884449 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:14.884622 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:14.884630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.885598 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.885603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:14.886556 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:14.886568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.887374 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.887380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:14.887574 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:14.888225 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:14.888254 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:14.891915 (+ 3661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":247,"threads_started":1}
W20260926 20:53:14.892360 18476 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:14.883232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:41781 (local address 127.8.249.190:46405)
0926 20:53:14.883751 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:14.883755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:14.883773 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:53:14.883895 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:14.883898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:14.883958 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:14.884045 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:14.884745 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.885455 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.886681 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:14.887198 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:14.887330 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:14.892109 (+ 4779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:41781: BlockingRecv error: recv got EOF from 127.8.249.130:41781 (error 108)
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":299,"threads_started":1}
Sep 26 20:53:15 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:15 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:15.274428 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.266074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:45107)
0926 20:53:15.266558 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.266573 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.266699 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.267012 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.267015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.267349 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:15.267503 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.267510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.268471 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.268474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.269374 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.269384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.270001 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.270004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.270115 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.270743 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:15.270760 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:15.274202 (+ 3442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":252,"threads_started":1}
W20260926 20:53:15.274587 18476 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:15.266242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:45107 (local address 127.8.249.190:46405)
0926 20:53:15.266451 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.266457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.266647 (+ 190us) server_negotiation.cc:407] Connection header received
0926 20:53:15.266784 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.266820 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.266891 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.266969 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:15.267634 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.268346 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.269486 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.269888 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.269976 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.274375 (+ 4399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:45107: BlockingRecv error: recv got EOF from 127.8.249.129:45107 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 20:53:15 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:15 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:15.287696 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.277684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:42571)
0926 20:53:15.278183 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.278199 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.278294 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.279491 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.279504 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.279880 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:15.280088 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.280103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.281492 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.281505 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.282405 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.282415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.283208 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.283216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.283393 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.283911 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:15.283928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:15.287373 (+ 3445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":304,"threads_started":1}
W20260926 20:53:15.287858 18476 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:15.278901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:42571 (local address 127.8.249.190:46405)
0926 20:53:15.279130 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.279136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.279154 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:53:15.279203 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.279207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.279317 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.279411 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:15.280243 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.281355 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.282563 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.283051 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.283176 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.287573 (+ 4397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:42571: BlockingRecv error: recv got EOF from 127.8.249.131:42571 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 20:53:15 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:15 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:15.902545 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:15.893196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:53193)
0926 20:53:15.893746 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:15.893770 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:15.893886 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:15.894337 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:15.894340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:15.894674 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:15.894938 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.894954 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.896028 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.896034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.896930 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:15.896940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.897699 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.897705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:15.897894 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.898578 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:15.898604 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:15.902176 (+ 3572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":265,"threads_started":1}
W20260926 20:53:15.902685 18480 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:15.893339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:53193 (local address 127.8.249.190:46405)
0926 20:53:15.894063 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:15.894067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:15.894083 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:53:15.894133 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:15.894136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:15.894201 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:15.894290 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:15.895136 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.895866 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.897089 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:15.897546 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:15.897678 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:15.902381 (+ 4703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:53193: BlockingRecv error: recv got EOF from 127.8.249.130:53193 (error 108)
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":505,"threads_started":1}
Sep 26 20:53:16 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:16 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:16.284864 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.275693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:50211)
0926 20:53:16.276268 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.276284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:16.276453 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:16.276840 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:16.276846 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:16.277274 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:16.277435 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:16.277445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.278553 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.278561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:16.279638 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:16.279652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.280342 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.280347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:16.280534 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:16.281142 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:16.281160 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:16.284507 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":324,"threads_started":1}
W20260926 20:53:16.284981 18480 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:16.275807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:50211 (local address 127.8.249.190:46405)
0926 20:53:16.276063 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:16.276071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:16.276383 (+ 312us) server_negotiation.cc:407] Connection header received
0926 20:53:16.276617 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:16.276621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:16.276685 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:16.276791 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:16.277598 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.278406 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.279791 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.280186 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.280317 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:16.284740 (+ 4423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:50211: BlockingRecv error: recv got EOF from 127.8.249.129:50211 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 26 20:53:16 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:16 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:16.298305 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.288617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:35423)
0926 20:53:16.289284 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.289301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:16.289474 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:16.289886 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:16.289891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:16.290299 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:16.290542 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:16.290573 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.291787 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.291794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:16.292820 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:16.292832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.293627 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.293634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:16.293851 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:16.294492 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:16.294510 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:16.297927 (+ 3417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":336,"threads_started":1}
W20260926 20:53:16.298377 18480 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:16.288766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:35423 (local address 127.8.249.190:46405)
0926 20:53:16.289063 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:16.289071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:16.289395 (+ 324us) server_negotiation.cc:407] Connection header received
0926 20:53:16.289628 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:16.289634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:16.289738 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:16.289860 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:16.290726 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.291602 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.292989 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.293466 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.293616 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:16.298135 (+ 4519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:35423: BlockingRecv error: recv got EOF from 127.8.249.131:35423 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Sep 26 20:53:16 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:16 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:16.912890 18483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:16.903517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:57541)
0926 20:53:16.904030 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:16.904060 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:16.904179 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:16.904499 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:16.904502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:16.904841 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:16.904997 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:16.905005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.905966 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.905970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:16.907029 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:16.907049 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.907947 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.907952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:16.908129 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:16.908787 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:16.908809 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:16.912645 (+ 3836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":248,"threads_started":1}
W20260926 20:53:16.913125 18484 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:16.903682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:57541 (local address 127.8.249.190:46405)
0926 20:53:16.904153 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:16.904157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:16.904173 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:53:16.904280 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:16.904283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:16.904352 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:16.904435 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:16.905120 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.905841 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.907172 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:16.907790 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:16.907920 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:16.912853 (+ 4933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:57541: BlockingRecv error: recv got EOF from 127.8.249.130:57541 (error 108)
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":269,"threads_started":1}
Sep 26 20:53:17 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:17 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:17.295487 18485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:17.286135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:32805)
0926 20:53:17.286667 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:17.286683 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:17.286878 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:17.287224 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:17.287229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:17.287635 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:17.287795 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:17.287803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.288838 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.288846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:17.289775 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:17.289786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.290472 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.290477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:17.290657 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:17.291333 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:17.291366 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:17.295259 (+ 3893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":289,"threads_started":1}
W20260926 20:53:17.295644 18484 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:17.286226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:32805 (local address 127.8.249.190:46405)
0926 20:53:17.286494 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:17.286500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:17.286778 (+ 278us) server_negotiation.cc:407] Connection header received
0926 20:53:17.287005 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:17.287012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:17.287080 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:17.287194 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:17.287935 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.288705 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.289921 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.290323 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.290458 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:17.295439 (+ 4981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:32805: BlockingRecv error: recv got EOF from 127.8.249.129:32805 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 26 20:53:17 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:17 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:17.308008 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:17.299337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:42213)
0926 20:53:17.299909 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:17.299933 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:17.300052 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:17.300349 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:17.300353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:17.300703 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:17.300855 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:17.300863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.301811 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.301814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:17.303039 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:17.303052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.303712 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.303716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:17.303835 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:17.304457 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:17.304474 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:17.307768 (+ 3294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":349,"threads_started":1}
W20260926 20:53:17.308243 18484 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:17.299403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:42213 (local address 127.8.249.190:46405)
0926 20:53:17.299580 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:17.299584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:17.300003 (+ 419us) server_negotiation.cc:407] Connection header received
0926 20:53:17.300149 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:17.300152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:17.300223 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:17.300306 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:17.300969 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.301690 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.303170 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.303546 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.303694 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:17.307948 (+ 4254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:42213: BlockingRecv error: recv got EOF from 127.8.249.131:42213 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 20:53:17 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:17 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:17.929579 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:17.914144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:39107)
0926 20:53:17.914651 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:17.914677 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:17.914827 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:17.915519 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:17.915523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:17.916093 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:17.916236 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:17.916245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.917294 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.917299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:17.918331 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:17.918343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.919136 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.919142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:17.919323 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:17.925246 (+ 5923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:17.925271 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:17.929353 (+ 4082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":242,"threads_started":1}
W20260926 20:53:17.929903 18488 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:17.914286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:39107 (local address 127.8.249.190:46405)
0926 20:53:17.914982 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:17.914987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:17.915005 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:53:17.915254 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:17.915261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:17.915365 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:17.915486 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:17.916374 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.917148 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.918484 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:17.918993 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:17.919125 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:17.929626 (+ 10501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:39107: BlockingRecv error: recv got EOF from 127.8.249.130:39107 (error 108)
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":438,"threads_started":1}
Sep 26 20:53:18 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:18 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:18.305697 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:18.296614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:43255)
0926 20:53:18.297170 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:18.297189 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:18.297334 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:18.297691 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:18.297697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:18.298130 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:18.298354 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:18.298367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.299460 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.299474 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:18.300376 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:18.300386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.301077 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.301083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:18.301257 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:18.301840 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:18.301860 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:18.305456 (+ 3596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":291,"threads_started":1}
W20260926 20:53:18.305881 18488 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:18.296684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:43255 (local address 127.8.249.190:46405)
0926 20:53:18.296884 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:18.296893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:18.297320 (+ 427us) server_negotiation.cc:407] Connection header received
0926 20:53:18.297452 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:18.297457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:18.297548 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:18.297661 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:18.298477 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.299323 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.300524 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.300905 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.301041 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:18.305663 (+ 4622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:43255: BlockingRecv error: recv got EOF from 127.8.249.129:43255 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 20:53:18 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:18 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:18.317925 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:18.308894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:33705)
0926 20:53:18.309409 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:18.309425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:18.309574 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:18.309913 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:18.309919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:18.310387 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:18.310579 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:18.310600 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.311960 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.311966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:18.313032 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:18.313044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.313701 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.313704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:18.313824 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:18.314344 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:18.314360 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:18.317650 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":270,"threads_started":1}
W20260926 20:53:18.318066 18488 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:18.308964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:33705 (local address 127.8.249.190:46405)
0926 20:53:18.309144 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:18.309150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:18.309505 (+ 355us) server_negotiation.cc:407] Connection header received
0926 20:53:18.309696 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:18.309705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:18.309780 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:18.309881 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:18.310721 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.311803 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.313159 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.313575 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.313675 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:18.317817 (+ 4142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:33705: BlockingRecv error: recv got EOF from 127.8.249.131:33705 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 20:53:18 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:18 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:18.939867 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:18.930555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:56269)
0926 20:53:18.931101 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:18.931123 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:18.931232 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:18.931510 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:18.931514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:18.931904 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:18.932056 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:18.932064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.933137 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.933141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:18.934161 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:18.934172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.935094 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.935100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:18.935295 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:18.936105 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:18.936132 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:18.939613 (+ 3481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":267,"threads_started":1}
W20260926 20:53:18.940156 18492 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:18.930686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:56269 (local address 127.8.249.190:46405)
0926 20:53:18.931173 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:18.931177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:18.931193 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:53:18.931335 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:18.931338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:18.931401 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:18.931485 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:18.932242 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.932979 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.934286 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:18.934892 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:18.935050 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:18.939826 (+ 4776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:56269: BlockingRecv error: recv got EOF from 127.8.249.130:56269 (error 108)
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":287,"threads_started":1}
Sep 26 20:53:19 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:19 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:19.315128 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:19.306669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:37113)
0926 20:53:19.307310 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:19.307328 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:19.307441 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:19.307733 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:19.307736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:19.308092 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:19.308251 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:19.308258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:19.309284 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:19.309287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:19.310317 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:19.310328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:19.311019 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:19.311023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:19.311148 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:19.311689 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:19.311707 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:19.314881 (+ 3174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":302,"threads_started":1}
W20260926 20:53:19.315203 18492 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:19.306756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:37113 (local address 127.8.249.190:46405)
0926 20:53:19.306972 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:19.306976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:19.307408 (+ 432us) server_negotiation.cc:407] Connection header received
0926 20:53:19.307539 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:19.307543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:19.307604 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:19.307696 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:19.308409 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:19.309169 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:19.310460 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:19.310882 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:19.311046 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:19.315041 (+ 3995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:37113: BlockingRecv error: recv got EOF from 127.8.249.129:37113 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 26 20:53:19 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:19 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:19.332541 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:19.318838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:57635)
0926 20:53:19.323211 (+ 4373us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:19.323232 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:19.323417 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:19.323794 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:19.323798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:19.324236 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:19.324401 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:19.324411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:19.325452 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:19.325464 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:19.326391 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:19.326402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:19.327229 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:19.327235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:19.327434 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:19.328075 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:19.328094 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:19.332210 (+ 4116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4263,"thread_start_us":4129,"threads_started":1}
W20260926 20:53:19.332660 18492 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:19.318906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:57635 (local address 127.8.249.190:46405)
0926 20:53:19.319082 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:19.319086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:19.323346 (+ 4260us) server_negotiation.cc:407] Connection header received
0926 20:53:19.323549 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:19.323555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:19.323647 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:19.323758 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:19.324560 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:19.325310 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:19.326541 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:19.327043 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:19.327187 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:19.332414 (+ 5227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:57635: BlockingRecv error: recv got EOF from 127.8.249.131:57635 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 20:53:19 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:19 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:19.950194 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:19.940880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:50301)
0926 20:53:19.941491 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:19.941511 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:19.941624 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:19.941986 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:19.941989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:19.942399 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:19.942565 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:19.942573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:19.943748 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:19.943751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:19.945112 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:19.945127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:19.945985 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:19.945991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:19.946152 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:19.946751 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:19.946768 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:19.949959 (+ 3191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":309,"threads_started":1}
W20260926 20:53:19.950400 18495 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:19.940946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:50301 (local address 127.8.249.190:46405)
0926 20:53:19.941523 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:19.941528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:19.941567 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:53:19.941770 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:19.941775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:19.941850 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:19.941950 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:19.942760 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:19.943619 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:19.945260 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:19.945774 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:19.945955 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:19.950152 (+ 4197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:50301: BlockingRecv error: recv got EOF from 127.8.249.130:50301 (error 108)
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":317,"threads_started":1}
Sep 26 20:53:20 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:20 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:20.325098 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:20.316121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:52255)
0926 20:53:20.316673 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:20.316688 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:20.316803 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:20.317121 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:20.317124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:20.317518 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:20.317675 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:20.317683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:20.319060 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:20.319065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:20.320248 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:20.320261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:20.320903 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:20.320908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:20.321023 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:20.321640 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:20.321657 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:20.324768 (+ 3111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":291,"threads_started":1}
W20260926 20:53:20.325176 18495 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:20.316246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:52255 (local address 127.8.249.190:46405)
0926 20:53:20.316421 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:20.316426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:20.316763 (+ 337us) server_negotiation.cc:407] Connection header received
0926 20:53:20.316899 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:20.316902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:20.316962 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:20.317070 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:20.317793 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:20.318909 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:20.320372 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:20.320753 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:20.320845 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:20.324939 (+ 4094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:52255: BlockingRecv error: recv got EOF from 127.8.249.129:52255 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 20:53:20 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:20 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:20.342466 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:20.333520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:45559)
0926 20:53:20.334202 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:20.334219 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:20.334323 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:20.334640 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:20.334646 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:20.335249 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:20.335539 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:20.335550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:20.336573 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:20.336577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:20.337504 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:20.337514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:20.338122 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:20.338126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:20.338231 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:20.338895 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:20.338912 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:20.342130 (+ 3218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":346,"threads_started":1}
W20260926 20:53:20.342576 18495 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:20.333712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:45559 (local address 127.8.249.190:46405)
0926 20:53:20.333855 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:20.333859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:20.334285 (+ 426us) server_negotiation.cc:407] Connection header received
0926 20:53:20.334407 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:20.334410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:20.334475 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:20.334550 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:20.335700 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:20.336458 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:20.337616 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:20.338001 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:20.338089 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:20.342315 (+ 4226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:45559: BlockingRecv error: recv got EOF from 127.8.249.131:45559 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 20:53:20 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:20 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:20.960536 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:20.951444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:52253)
0926 20:53:20.952323 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:20.952350 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:20.952478 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:20.952764 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:20.952767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:20.953175 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:20.953333 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:20.953340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:20.954479 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:20.954483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:20.955476 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:20.955488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:20.956153 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:20.956157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:20.956278 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:20.956960 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:20.956978 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:20.960281 (+ 3303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":282,"threads_started":1}
W20260926 20:53:20.960669 18500 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:20.951522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:52253 (local address 127.8.249.190:46405)
0926 20:53:20.952254 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:20.952261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:20.952417 (+ 156us) server_negotiation.cc:407] Connection header received
0926 20:53:20.952577 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:20.952581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:20.952639 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:20.952722 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:20.953450 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:20.954330 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:20.955596 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:20.955995 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:20.956093 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:20.960470 (+ 4377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:52253: BlockingRecv error: recv got EOF from 127.8.249.130:52253 (error 108)
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":450,"threads_started":1}
Sep 26 20:53:21 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:21 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:21.335090 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:21.326094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:37535)
0926 20:53:21.326583 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:21.326598 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:21.326700 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:21.327054 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:21.327057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:21.327445 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:21.327600 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:21.327608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:21.328586 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:21.328589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:21.329537 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:21.329547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:21.330179 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:21.330183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:21.330308 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:21.330929 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:21.330945 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:21.334753 (+ 3808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":291,"threads_started":1}
W20260926 20:53:21.335249 18500 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:21.326178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:37535 (local address 127.8.249.190:46405)
0926 20:53:21.326413 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:21.326419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:21.326677 (+ 258us) server_negotiation.cc:407] Connection header received
0926 20:53:21.326871 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:21.326875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:21.326936 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:21.327012 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:21.327728 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:21.328479 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:21.329685 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:21.330059 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:21.330149 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:21.335014 (+ 4865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:37535: BlockingRecv error: recv got EOF from 127.8.249.129:37535 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 20:53:21 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:21 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:21.352634 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:21.343559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:41243)
0926 20:53:21.344133 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:21.344148 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:21.344275 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:21.344886 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:21.344891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:21.345353 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:21.345533 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:21.345543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:21.346731 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:21.346735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:21.347808 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:21.347820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:21.348469 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:21.348473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:21.348587 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:21.349113 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:21.349129 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:21.352391 (+ 3262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":294,"threads_started":1}
W20260926 20:53:21.352885 18500 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:21.343634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:41243 (local address 127.8.249.190:46405)
0926 20:53:21.343816 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:21.343828 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:21.344270 (+ 442us) server_negotiation.cc:407] Connection header received
0926 20:53:21.344635 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:21.344642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:21.344732 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:21.344824 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:21.345698 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:21.346607 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:21.347928 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:21.348321 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:21.348421 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:21.352612 (+ 4191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:41243: BlockingRecv error: recv got EOF from 127.8.249.131:41243 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 20:53:21 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:21 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:21.970670 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:21.961810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:56745)
0926 20:53:21.962327 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:21.962348 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:21.962448 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:21.962742 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:21.962745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:21.963160 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:21.963295 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:21.963303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:21.964245 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:21.964248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:21.965178 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:21.965187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:21.965984 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:21.965988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:21.966127 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:21.966697 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:21.966713 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:21.970446 (+ 3733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":280,"threads_started":1}
W20260926 20:53:21.971062 18504 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:21.961895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:56745 (local address 127.8.249.190:46405)
0926 20:53:21.962457 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:21.962461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:21.962478 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:53:21.962530 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:21.962533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:21.962597 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:21.962693 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:21.963421 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:21.964131 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:21.965341 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:21.965837 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:21.965947 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:21.970676 (+ 4729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:56745: BlockingRecv error: recv got EOF from 127.8.249.130:56745 (error 108)
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":331,"threads_started":1}
Sep 26 20:53:22 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:22 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:22.344995 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:22.336115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:51895)
0926 20:53:22.336660 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:22.336675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:22.336829 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:22.337263 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:22.337267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:22.337793 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:22.338027 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:22.338041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:22.339096 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:22.339100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:22.340054 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:22.340064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:22.340668 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:22.340672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:22.340780 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:22.341350 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:22.341366 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:22.344749 (+ 3383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":306,"threads_started":1}
W20260926 20:53:22.345224 18504 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:22.336238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:51895 (local address 127.8.249.190:46405)
0926 20:53:22.336472 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:22.336478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:22.336761 (+ 283us) server_negotiation.cc:407] Connection header received
0926 20:53:22.336999 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:22.337004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:22.337105 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:22.337194 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:22.338178 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:22.338983 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:22.340168 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:22.340547 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:22.340637 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:22.344964 (+ 4327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:51895: BlockingRecv error: recv got EOF from 127.8.249.129:51895 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 20:53:22 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:22 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:22.362880 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:22.353525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:50569)
0926 20:53:22.354018 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:22.354035 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:22.354153 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:22.354457 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:22.354460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:22.354993 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:22.355152 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:22.355161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:22.356152 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:22.356156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:22.357104 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:22.357116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:22.357795 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:22.357800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:22.357942 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:22.358625 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:22.358642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:22.362535 (+ 3893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":269,"threads_started":1}
W20260926 20:53:22.363229 18504 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:22.353637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:50569 (local address 127.8.249.190:46405)
0926 20:53:22.353881 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:22.353887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:22.354120 (+ 233us) server_negotiation.cc:407] Connection header received
0926 20:53:22.354240 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:22.354243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:22.354315 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:22.354395 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:22.355284 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:22.356011 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:22.357241 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:22.357658 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:22.357748 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:22.362927 (+ 5179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:50569: BlockingRecv error: recv got EOF from 127.8.249.131:50569 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 26 20:53:22 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:22 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:22.984341 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:22.971598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:45923)
0926 20:53:22.972056 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:22.972076 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:22.972179 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:22.972694 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:22.972698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:22.973048 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:22.973205 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:22.973213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:22.974262 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:22.974265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:22.975294 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:22.975305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:22.976001 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:22.976005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:22.976107 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:22.980348 (+ 4241us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:22.980368 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:22.983903 (+ 3535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":248,"threads_started":1}
W20260926 20:53:22.984499 18508 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:22.971697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:45923 (local address 127.8.249.190:46405)
0926 20:53:22.972440 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:22.972445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:22.972460 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:53:22.972504 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:22.972507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:22.972565 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:22.972653 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:22.973375 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:22.974143 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:22.975431 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:22.975863 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:22.975964 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:22.984184 (+ 8220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:45923: BlockingRecv error: recv got EOF from 127.8.249.130:45923 (error 108)
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":488,"threads_started":1}
Sep 26 20:53:23 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:23 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:23.355739 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:23.346096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:42895)
0926 20:53:23.346637 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:23.346653 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:23.346774 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:23.347219 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:23.347226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:23.347811 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:23.348052 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:23.348066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:23.349464 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:23.349468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:23.350381 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:23.350393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:23.351152 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:23.351157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:23.351285 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:23.352000 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:23.352019 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:23.355506 (+ 3487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":313,"threads_started":1}
W20260926 20:53:23.355983 18508 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:23.346267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:42895 (local address 127.8.249.190:46405)
0926 20:53:23.346512 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:23.346516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:23.346734 (+ 218us) server_negotiation.cc:407] Connection header received
0926 20:53:23.346911 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:23.346915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:23.346983 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:23.347112 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:23.348233 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:23.349323 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:23.350516 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:23.351019 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:23.351125 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:23.355733 (+ 4608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:42895: BlockingRecv error: recv got EOF from 127.8.249.129:42895 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 26 20:53:23 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:23 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:23.374431 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:23.363966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:38227)
0926 20:53:23.364517 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:23.364540 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:23.364699 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:23.365106 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:23.365112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:23.365619 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:23.365847 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:23.365861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:23.367384 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:23.367391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:23.368871 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:23.368885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:23.369568 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:23.369572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:23.369691 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:23.370306 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:23.370324 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:23.374186 (+ 3862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":288,"threads_started":1}
W20260926 20:53:23.374598 18508 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:23.364038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:38227 (local address 127.8.249.190:46405)
0926 20:53:23.364287 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:23.364293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:23.364642 (+ 349us) server_negotiation.cc:407] Connection header received
0926 20:53:23.364834 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:23.364841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:23.364947 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:23.365045 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:23.366014 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:23.367220 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:23.369014 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:23.369409 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:23.369516 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:23.374373 (+ 4857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:38227: BlockingRecv error: recv got EOF from 127.8.249.131:38227 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 20:53:23 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:23 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:23.994459 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:23.985357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:57437)
0926 20:53:23.985844 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:23.985866 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:23.985972 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:23.986443 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:23.986447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:23.986933 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:23.987111 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:23.987120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:23.988112 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:23.988116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:23.989027 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:23.989038 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:23.989672 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:23.989676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:23.989809 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:23.990365 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:23.990382 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:23.994122 (+ 3740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":267,"threads_started":1}
W20260926 20:53:23.994616 18512 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:23.985523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:57437 (local address 127.8.249.190:46405)
0926 20:53:23.986134 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:23.986138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:23.986152 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:53:23.986194 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:23.986197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:23.986277 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:23.986363 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:23.987237 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:23.987996 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:23.989149 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:23.989551 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:23.989641 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:23.994311 (+ 4670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:57437: BlockingRecv error: recv got EOF from 127.8.249.130:57437 (error 108)
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":404,"threads_started":1}
Sep 26 20:53:24 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:24 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:24.367982 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:24.356965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:42191)
0926 20:53:24.358057 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:24.358073 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:24.358234 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:24.358640 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:24.358645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:24.359212 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:24.359383 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:24.359392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:24.360884 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:24.360888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:24.361835 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:24.361847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:24.362870 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:24.362877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:24.363021 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:24.363661 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:24.363679 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:24.367644 (+ 3965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":995,"thread_start_us":846,"threads_started":1}
W20260926 20:53:24.368109 18512 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:24.357230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:42191 (local address 127.8.249.190:46405)
0926 20:53:24.357578 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:24.357585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:24.358177 (+ 592us) server_negotiation.cc:407] Connection header received
0926 20:53:24.358391 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:24.358397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:24.358476 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:24.358587 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:24.359528 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:24.360717 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:24.361972 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:24.362625 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:24.362748 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:24.367878 (+ 5130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:42191: BlockingRecv error: recv got EOF from 127.8.249.129:42191 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 26 20:53:24 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:24 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:24.384212 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:24.375402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:37435)
0926 20:53:24.375932 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:24.375950 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:24.376056 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:24.376366 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:24.376370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:24.376739 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:24.376913 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:24.376921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:24.377807 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:24.377810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:24.378704 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:24.378713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:24.379441 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:24.379445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:24.379560 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:24.380196 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:24.380213 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:24.383992 (+ 3779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":315,"threads_started":1}
W20260926 20:53:24.384522 18512 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:24.375513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:37435 (local address 127.8.249.190:46405)
0926 20:53:24.375683 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:24.375687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:24.376017 (+ 330us) server_negotiation.cc:407] Connection header received
0926 20:53:24.376155 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:24.376159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:24.376229 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:24.376311 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:24.377020 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:24.377702 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:24.378887 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:24.379317 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:24.379407 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:24.384159 (+ 4752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:37435: BlockingRecv error: recv got EOF from 127.8.249.131:37435 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 20:53:24 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:24 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:25.004459 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:24.995421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:55197)
0926 20:53:24.995942 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:24.995962 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:24.996074 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:24.996375 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:24.996378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:24.996731 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:24.996890 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:24.996898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:24.997901 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:24.997905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:24.998882 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:24.998895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:24.999708 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:24.999712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:24.999936 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:25.000663 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:25.000680 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:25.004221 (+ 3541us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":294,"threads_started":1}
W20260926 20:53:25.004671 18516 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:24.995492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:55197 (local address 127.8.249.190:46405)
0926 20:53:24.995962 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:24.995966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:24.996014 (+ 48us) server_negotiation.cc:407] Connection header received
0926 20:53:24.996173 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:24.996176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:24.996248 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:24.996338 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:24.997036 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:24.997790 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:24.999013 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:24.999561 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:24.999697 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:25.004430 (+ 4733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:55197: BlockingRecv error: recv got EOF from 127.8.249.130:55197 (error 108)
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":287,"threads_started":1}
Sep 26 20:53:25 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:25 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:25.377998 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:25.369148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:58935)
0926 20:53:25.369633 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:25.369650 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:25.369765 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:25.370051 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:25.370054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:25.370430 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:25.370569 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:25.370577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:25.371706 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:25.371710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:25.372702 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:25.372756 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:25.373524 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:25.373528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:25.373648 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:25.374225 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:25.374244 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:25.377745 (+ 3501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":265,"threads_started":1}
W20260926 20:53:25.378141 18516 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:25.369213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:58935 (local address 127.8.249.190:46405)
0926 20:53:25.369386 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:25.369390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:25.369722 (+ 332us) server_negotiation.cc:407] Connection header received
0926 20:53:25.369857 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:25.369861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:25.369928 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:25.370006 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:25.370685 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:25.371502 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:25.372954 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:25.373393 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:25.373486 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:25.377946 (+ 4460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:58935: BlockingRecv error: recv got EOF from 127.8.249.129:58935 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 20:53:25 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:25 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:25.394954 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:25.385235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:33639)
0926 20:53:25.385760 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:25.385778 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:25.385937 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:25.386303 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:25.386310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:25.386682 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:25.386901 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:25.386911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:25.387918 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:25.387923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:25.389000 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:25.389012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:25.389718 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:25.389723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:25.389909 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:25.390530 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:25.390547 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:25.394626 (+ 4079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":288,"threads_started":1}
W20260926 20:53:25.395144 18516 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:25.385326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:33639 (local address 127.8.249.190:46405)
0926 20:53:25.385578 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:25.385586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:25.385865 (+ 279us) server_negotiation.cc:407] Connection header received
0926 20:53:25.386071 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:25.386074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:25.386145 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:25.386246 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:25.387039 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:25.387770 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:25.389141 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:25.389528 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:25.389660 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:25.394901 (+ 5241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:33639: BlockingRecv error: recv got EOF from 127.8.249.131:33639 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 26 20:53:26 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:26 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.130@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:26.015090 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:26.005506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.130:51959)
0926 20:53:26.006037 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:26.006060 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:26.006166 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:26.006479 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:26.006482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:26.006955 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:26.007099 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:26.007107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.008260 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.008265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:26.009236 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:26.009248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.009992 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.009997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:26.010203 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:26.010957 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:26.010980 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:26.014861 (+ 3881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":300,"threads_started":1}
W20260926 20:53:26.015364 18520 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:26.005667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:51959 (local address 127.8.249.190:46405)
0926 20:53:26.006171 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:26.006176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:26.006207 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:53:26.006260 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:26.006264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:26.006335 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:26.006442 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:26.007270 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.008106 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.009427 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.009834 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.009934 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:26.015060 (+ 5126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.130:51959: BlockingRecv error: recv got EOF from 127.8.249.130:51959 (error 108)
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":289,"threads_started":1}
Sep 26 20:53:26 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:26 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.129@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:26.388494 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:26.379495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.129:48883)
0926 20:53:26.380008 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:26.380025 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:26.380149 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:26.380458 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:26.380461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:26.380801 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:26.380959 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:26.380967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.382034 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.382040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:26.383058 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:26.383069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.383733 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.383739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:26.383926 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:26.384547 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:26.384564 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:26.388093 (+ 3529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":282,"threads_started":1}
W20260926 20:53:26.388540 18520 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:26.379600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:48883 (local address 127.8.249.190:46405)
0926 20:53:26.379863 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:26.379871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:26.380124 (+ 253us) server_negotiation.cc:407] Connection header received
0926 20:53:26.380260 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:26.380264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:26.380330 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:26.380425 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:26.381112 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.381889 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.383193 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.383591 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.383757 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:26.388308 (+ 4551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.129:48883: BlockingRecv error: recv got EOF from 127.8.249.129:48883 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 26 20:53:26 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:53:26 dist-test-slave-2dg3 krb5kdc[17462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.249.131@KRBTEST.COM for oryx/127.8.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:53:26.404575 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:26.395858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46405 (local address 127.8.249.131:43927)
0926 20:53:26.396386 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:26.396402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:26.396519 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:26.396891 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:26.396894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:26.397282 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:26.397425 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:26.397433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.398569 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.398573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:26.399663 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:26.399676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.400354 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.400358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:26.400479 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:26.401050 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:26.401068 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:26.404347 (+ 3279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.249.190:46405: Server oryx/127.8.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":268,"threads_started":1}
W20260926 20:53:26.404742 18520 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:26.395942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:43927 (local address 127.8.249.190:46405)
0926 20:53:26.396282 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:26.396289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:26.396481 (+ 192us) server_negotiation.cc:407] Connection header received
0926 20:53:26.396665 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:26.396669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:26.396761 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:26.396854 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:26.397547 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.398287 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.399821 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.400210 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.400313 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:26.404521 (+ 4208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.249.131:43927: BlockingRecv error: recv got EOF from 127.8.249.131:43927 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
I20260926 20:53:26.934633 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 18050
I20260926 20:53:26.947110 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:26.935947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.8.249.129:34527)
0926 20:53:26.936551 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:26.936557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:26.936583 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:53:26.936649 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:26.936656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:26.936898 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:26.937077 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:26.938464 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.939624 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.940850 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.941437 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.941548 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:26.944036 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:26.944045 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:26.944060 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:26.944110 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:26.946489 (+ 2379us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.249.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 20:53:26.946879 (+ 390us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41936: Request ticket server kudu/127.8.249.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":299,"threads_started":1}
W20260926 20:53:26.947247 18524 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41936: Request ticket server kudu/127.8.249.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 20:53:26.947232 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 18050: Not authorized: Client connection negotiation failed: client connection to 127.8.249.129:34527: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.249.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 20:53:26.947589 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 18050
I20260926 20:53:26.964412 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 18201
I20260926 20:53:26.977054 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:26.965560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.8.249.130:35849)
0926 20:53:26.966232 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:26.966241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:26.966273 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:53:26.966349 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:26.966372 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:26.966676 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:26.966932 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:26.968155 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.969125 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.970462 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.971144 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:26.971293 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:26.973792 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:26.973800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:26.973814 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:26.973851 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:26.976319 (+ 2468us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.249.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 20:53:26.976705 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44808: Request ticket server kudu/127.8.249.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":335,"threads_started":1}
W20260926 20:53:26.977113 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 18201: Not authorized: Client connection negotiation failed: client connection to 127.8.249.130:35849: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.249.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 20:53:26.977171 18525 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44808: Request ticket server kudu/127.8.249.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 20:53:26.977295 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 18201
I20260926 20:53:26.994136 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 18333
I20260926 20:53:27.007175 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:26.995353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.8.249.131:39685)
0926 20:53:26.996107 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:26.996114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:26.996141 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:53:26.996210 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:26.996217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:26.996448 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:26.996668 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:26.998007 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:26.999075 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:27.000317 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:27.000886 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:27.001033 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:27.003765 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:27.003774 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:27.003789 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:27.003833 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:27.006405 (+ 2572us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.249.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 20:53:27.006838 (+ 433us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40420: Request ticket server kudu/127.8.249.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":430,"threads_started":1}
W20260926 20:53:27.007189 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 18333: Not authorized: Client connection negotiation failed: client connection to 127.8.249.131:39685: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.249.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 20:53:27.007289 18526 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40420: Request ticket server kudu/127.8.249.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 20:53:27.007361 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 18333
I20260926 20:53:27.023694 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 17981
I20260926 20:53:27.025183 18043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:27.141126 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 17981
2026-09-26T20:53:27Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21242 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T20:53:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:53:27Z Disabled control of system clock
I20260926 20:53:27.223440 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:44575
--webserver_interface=127.8.249.190
--webserver_port=0
--builtin_ntp_servers=127.8.249.148:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:53:27.554085 18531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:27.554410 18531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:27.554490 18531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:27.561980 18531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:27.562111 18531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:27.562150 18531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:27.562199 18531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:27.562244 18531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:27.571848 18531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:44575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:27.573117 18531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:27.574780 18531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:27.587049 18536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:27.587116 18537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:27.587080 18539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:27.587687 18531 server_base.cc:1034] running on GCE node
I20260926 20:53:27.588522 18531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:27.589978 18531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:27.591216 18531 hybrid_clock.cc:648] HybridClock initialized: now 1790456007591175 us; error 55 us; skew 500 ppm
I20260926 20:53:27.591614 18531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:53:27.594038 18531 webserver.cc:460] Webserver started at http://127.8.249.190:43043/ using document root <none> and password file <none>
I20260926 20:53:27.594767 18531 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:27.594944 18531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:27.595278 18531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:27.598227 18531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "5385c861adcd4c93852a1bc22d06ab56"
format_stamp: "Formatted at 2026-09-26 20:53:27 on dist-test-slave-2dg3"
server_key: "4bb8e6b35dbbadafc979c92dd0c0bb4d"
server_key_iv: "da36e7d16d4fbff0d6c336520f0bf206"
server_key_version: "encryptionkey@0"
I20260926 20:53:27.599043 18531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "5385c861adcd4c93852a1bc22d06ab56"
format_stamp: "Formatted at 2026-09-26 20:53:27 on dist-test-slave-2dg3"
server_key: "4bb8e6b35dbbadafc979c92dd0c0bb4d"
server_key_iv: "da36e7d16d4fbff0d6c336520f0bf206"
server_key_version: "encryptionkey@0"
I20260926 20:53:27.604095 18531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:53:27.607370 18545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:27.608744 18531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:53:27.608944 18531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "5385c861adcd4c93852a1bc22d06ab56"
format_stamp: "Formatted at 2026-09-26 20:53:27 on dist-test-slave-2dg3"
server_key: "4bb8e6b35dbbadafc979c92dd0c0bb4d"
server_key_iv: "da36e7d16d4fbff0d6c336520f0bf206"
server_key_version: "encryptionkey@0"
I20260926 20:53:27.609177 18531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:27.626214 18531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:27.627167 18531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:27.627483 18531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:27.645677 18531 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:44575
I20260926 20:53:27.645700 18596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:44575 every 8 connection(s)
I20260926 20:53:27.647249 18531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:27.651081 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:27.656970 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 18531
I20260926 20:53:27.657191 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:53:27.657830 9190 external_mini_cluster.cc:1428] Setting key 6192cc9977918785e353e307faea9167
I20260926 20:53:27.662778 18597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56: Bootstrap starting.
I20260926 20:53:27.666986 18597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:27.668290 18597 log.cc:824] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:27.671931 18597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56: No bootstrap required, opened a new log
I20260926 20:53:27.673992 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:27.660099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.8.249.190:44575)
0926 20:53:27.661252 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:27.661269 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:27.661328 (+ 59us) server_negotiation.cc:407] Connection header received
0926 20:53:27.663722 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:27.663753 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:27.664369 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:27.664897 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:27.665470 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:27.667477 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:27.668932 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:27.669783 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:27.669980 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:27.671125 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:27.671136 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:27.671156 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:27.671887 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:27.672007 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:27.672034 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:27.672150 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:27.672338 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 20:53:27.672700 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":357,"threads_started":1}
I20260926 20:53:27.680541 18597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5385c861adcd4c93852a1bc22d06ab56" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44575 } }
I20260926 20:53:27.680898 18597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:27.680980 18597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5385c861adcd4c93852a1bc22d06ab56, State: Initialized, Role: FOLLOWER
I20260926 20:53:27.681565 18597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5385c861adcd4c93852a1bc22d06ab56" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44575 } }
I20260926 20:53:27.681852 18597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:53:27.682041 18597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:53:27.682191 18597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:27.684505 18597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5385c861adcd4c93852a1bc22d06ab56" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44575 } }
I20260926 20:53:27.685019 18597 leader_election.cc:302] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5385c861adcd4c93852a1bc22d06ab56; no voters:
I20260926 20:53:27.685600 18597 leader_election.cc:288] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:53:27.685768 18602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:27.687201 18602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [term 1 LEADER]: Becoming Leader. State: Replica: 5385c861adcd4c93852a1bc22d06ab56, State: Running, Role: LEADER
I20260926 20:53:27.687803 18602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5385c861adcd4c93852a1bc22d06ab56" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44575 } }
I20260926 20:53:27.687954 18597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:27.690652 18604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5385c861adcd4c93852a1bc22d06ab56. Latest consensus state: current_term: 1 leader_uuid: "5385c861adcd4c93852a1bc22d06ab56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5385c861adcd4c93852a1bc22d06ab56" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44575 } } }
I20260926 20:53:27.690864 18603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5385c861adcd4c93852a1bc22d06ab56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5385c861adcd4c93852a1bc22d06ab56" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44575 } } }
I20260926 20:53:27.691213 18603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:27.691099 18604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56 [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:27.692998 18611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:27.698731 18611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:53:27.707401 18611 catalog_manager.cc:1347] Generated new cluster ID: c9e778afbdc04218982afa0c1d01b83e
I20260926 20:53:27.707541 18611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:53:27.732059 18611 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:53:27.733641 18611 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:53:27.740494 18611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5385c861adcd4c93852a1bc22d06ab56: Generated new TSK 0
I20260926 20:53:27.741410 18611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:53:27.750017 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44575
--builtin_ntp_servers=127.8.249.148:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 20:53:28.088907 18621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:28.089313 18621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:28.089463 18621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:28.097039 18621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:28.097146 18621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:28.097324 18621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:53:28.106225 18621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:28.107592 18621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:28.109267 18621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:28.122144 18626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:28.122228 18629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:28.122182 18627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:28.122964 18621 server_base.cc:1034] running on GCE node
I20260926 20:53:28.123613 18621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:28.124574 18621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:28.125805 18621 hybrid_clock.cc:648] HybridClock initialized: now 1790456008125742 us; error 54 us; skew 500 ppm
I20260926 20:53:28.126171 18621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:53:28.135010 18621 webserver.cc:460] Webserver started at http://127.8.249.129:44907/ using document root <none> and password file <none>
I20260926 20:53:28.135795 18621 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:28.135926 18621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:28.136273 18621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:28.139263 18621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "9da8a98043e04896b229e657074c1e1b"
format_stamp: "Formatted at 2026-09-26 20:53:28 on dist-test-slave-2dg3"
server_key: "17e93a299f1be542f11b9641aec3c293"
server_key_iv: "8a5966cecc4e5b00a624f3408c8704bc"
server_key_version: "encryptionkey@0"
I20260926 20:53:28.140072 18621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "9da8a98043e04896b229e657074c1e1b"
format_stamp: "Formatted at 2026-09-26 20:53:28 on dist-test-slave-2dg3"
server_key: "17e93a299f1be542f11b9641aec3c293"
server_key_iv: "8a5966cecc4e5b00a624f3408c8704bc"
server_key_version: "encryptionkey@0"
I20260926 20:53:28.145251 18621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:53:28.148679 18635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:28.150025 18621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:53:28.150202 18621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "9da8a98043e04896b229e657074c1e1b"
format_stamp: "Formatted at 2026-09-26 20:53:28 on dist-test-slave-2dg3"
server_key: "17e93a299f1be542f11b9641aec3c293"
server_key_iv: "8a5966cecc4e5b00a624f3408c8704bc"
server_key_version: "encryptionkey@0"
I20260926 20:53:28.150413 18621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:28.166651 18621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:28.167565 18621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:28.167893 18621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:28.168850 18621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:28.170452 18621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:28.170540 18621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:28.170652 18621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:28.170751 18621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:28.204842 18621 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:36199
I20260926 20:53:28.204921 18747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:36199 every 8 connection(s)
I20260926 20:53:28.206161 18621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:53:28.215297 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 18621
I20260926 20:53:28.215487 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:53:28.216104 9190 external_mini_cluster.cc:1428] Setting key 3dc31003b531cf68db31bc6b84e9e8b9
I20260926 20:53:28.218783 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44575
--builtin_ntp_servers=127.8.249.148:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 20:53:28.219558 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:28.209028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:50249 (local address 127.8.249.190:44575)
0926 20:53:28.209528 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:28.209536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:28.210835 (+ 1299us) server_negotiation.cc:407] Connection header received
0926 20:53:28.212581 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:28.212585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:28.212663 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:28.212787 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:28.213960 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:28.214733 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:28.216579 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:28.217097 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:28.217256 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:28.218742 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:28.218753 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:28.218756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:28.218774 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:28.218866 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:28.218880 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:28.218992 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:28.219245 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 20:53:28.219320 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":228,"threads_started":1}
I20260926 20:53:28.221114 18751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:28.209727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44575 (local address 127.8.249.129:50249)
0926 20:53:28.210641 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:28.210676 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:28.212302 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:28.213010 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:28.213019 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:28.213114 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:28.213771 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:28.213788 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:28.214947 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:28.214956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:28.216421 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:28.216433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:28.217348 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:28.217357 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:28.217597 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:28.218423 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:53:28.218426 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:28.218465 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:53:28.218544 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:28.219011 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:28.219025 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:28.219037 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 20:53:28.219226 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 20:53:28.219667 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":289,"threads_started":1}
I20260926 20:53:28.222669 18748 heartbeater.cc:347] Connected to a master server at 127.8.249.190:44575
I20260926 20:53:28.223014 18748 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:28.223984 18748 heartbeater.cc:511] Master 127.8.249.190:44575 requested a full tablet report, sending...
I20260926 20:53:28.226063 18562 ts_manager.cc:194] Registered new tserver with Master: 9da8a98043e04896b229e657074c1e1b (127.8.249.129:36199)
I20260926 20:53:28.228237 18562 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.249.129:50249
W20260926 20:53:28.544335 18752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:28.544626 18752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:28.544709 18752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:28.551982 18752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:28.552089 18752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:28.552199 18752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:53:28.561276 18752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:28.562598 18752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:28.564458 18752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:28.578406 18757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:28.578474 18760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:28.578450 18758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:28.579272 18752 server_base.cc:1034] running on GCE node
I20260926 20:53:28.579892 18752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:28.580945 18752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:28.582150 18752 hybrid_clock.cc:648] HybridClock initialized: now 1790456008582081 us; error 59 us; skew 500 ppm
I20260926 20:53:28.582497 18752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:53:28.584942 18752 webserver.cc:460] Webserver started at http://127.8.249.130:45663/ using document root <none> and password file <none>
I20260926 20:53:28.585685 18752 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:28.585806 18752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:28.586148 18752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:28.589353 18752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "dac644ab045a4cfdbe3017535175c616"
format_stamp: "Formatted at 2026-09-26 20:53:28 on dist-test-slave-2dg3"
server_key: "ebdf2d90c3e24ba08998e87b3c4d6fef"
server_key_iv: "59dc97f574f912d90d43b8ab20fd4045"
server_key_version: "encryptionkey@0"
I20260926 20:53:28.590163 18752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "dac644ab045a4cfdbe3017535175c616"
format_stamp: "Formatted at 2026-09-26 20:53:28 on dist-test-slave-2dg3"
server_key: "ebdf2d90c3e24ba08998e87b3c4d6fef"
server_key_iv: "59dc97f574f912d90d43b8ab20fd4045"
server_key_version: "encryptionkey@0"
I20260926 20:53:28.595546 18752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 20:53:28.598997 18766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:28.600291 18752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:53:28.600471 18752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "dac644ab045a4cfdbe3017535175c616"
format_stamp: "Formatted at 2026-09-26 20:53:28 on dist-test-slave-2dg3"
server_key: "ebdf2d90c3e24ba08998e87b3c4d6fef"
server_key_iv: "59dc97f574f912d90d43b8ab20fd4045"
server_key_version: "encryptionkey@0"
I20260926 20:53:28.600687 18752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:28.615970 18752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:28.616892 18752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:28.617218 18752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:28.618201 18752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:28.619697 18752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:28.619784 18752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:28.619942 18752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:28.620033 18752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:28.651121 18752 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:36287
I20260926 20:53:28.651207 18878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:36287 every 8 connection(s)
I20260926 20:53:28.652412 18752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:53:28.653532 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 18752
I20260926 20:53:28.653718 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:53:28.654430 9190 external_mini_cluster.cc:1428] Setting key c1f507bae9c8618aa3b2c251166745c5
I20260926 20:53:28.658538 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44575
--builtin_ntp_servers=127.8.249.148:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 20:53:28.678377 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:28.655059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:33089 (local address 127.8.249.190:44575)
0926 20:53:28.655235 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:28.655241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:28.667952 (+ 12711us) server_negotiation.cc:407] Connection header received
0926 20:53:28.671602 (+ 3650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:28.671606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:28.671668 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:28.671752 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:28.672857 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:28.673569 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:28.675198 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:28.675681 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:28.675858 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:28.677395 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:28.677406 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:28.677410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:28.677435 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:28.677471 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:28.677494 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:28.677656 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:28.677901 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 20:53:28.678007 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:53:28.679520 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:28.656193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44575 (local address 127.8.249.130:33089)
0926 20:53:28.667786 (+ 11593us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:28.667833 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:28.671291 (+ 3458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:28.672011 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:28.672022 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:28.672121 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:28.672691 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:28.672706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:28.673719 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:28.673725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:28.675051 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:28.675062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:28.675944 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:28.675950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:28.676178 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:28.676921 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:53:28.676924 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:28.676964 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:53:28.677040 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:28.677641 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:28.677653 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:28.677663 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:53:28.677877 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 20:53:28.678301 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10926,"thread_start_us":11672,"threads_started":1}
I20260926 20:53:28.681262 18879 heartbeater.cc:347] Connected to a master server at 127.8.249.190:44575
I20260926 20:53:28.681593 18879 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:28.682391 18879 heartbeater.cc:511] Master 127.8.249.190:44575 requested a full tablet report, sending...
I20260926 20:53:28.683966 18562 ts_manager.cc:194] Registered new tserver with Master: dac644ab045a4cfdbe3017535175c616 (127.8.249.130:36287)
I20260926 20:53:28.685278 18562 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.249.130:33089
W20260926 20:53:29.005894 18882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:29.006232 18882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:29.006351 18882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:29.014693 18882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:29.014843 18882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:29.015033 18882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:53:29.032051 18882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:44575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:29.033840 18882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:29.036273 18882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:29.054041 18888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:29.054067 18887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:29.054391 18890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:29.055291 18882 server_base.cc:1034] running on GCE node
I20260926 20:53:29.056020 18882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:29.057071 18882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:29.058275 18882 hybrid_clock.cc:648] HybridClock initialized: now 1790456009058208 us; error 61 us; skew 500 ppm
I20260926 20:53:29.058611 18882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:53:29.061379 18882 webserver.cc:460] Webserver started at http://127.8.249.131:45455/ using document root <none> and password file <none>
I20260926 20:53:29.062327 18882 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:29.062469 18882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:29.062876 18882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:29.066551 18882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "0a053f62b8d34c80810a523c041850ee"
format_stamp: "Formatted at 2026-09-26 20:53:29 on dist-test-slave-2dg3"
server_key: "abedf81c3e55f8bc42f1b872c22e2a61"
server_key_iv: "dca6d877487c640039fe5cb58bdf20bb"
server_key_version: "encryptionkey@0"
I20260926 20:53:29.067502 18882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "0a053f62b8d34c80810a523c041850ee"
format_stamp: "Formatted at 2026-09-26 20:53:29 on dist-test-slave-2dg3"
server_key: "abedf81c3e55f8bc42f1b872c22e2a61"
server_key_iv: "dca6d877487c640039fe5cb58bdf20bb"
server_key_version: "encryptionkey@0"
I20260926 20:53:29.072635 18882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:53:29.076277 18896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:29.077651 18882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:53:29.077847 18882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "0a053f62b8d34c80810a523c041850ee"
format_stamp: "Formatted at 2026-09-26 20:53:29 on dist-test-slave-2dg3"
server_key: "abedf81c3e55f8bc42f1b872c22e2a61"
server_key_iv: "dca6d877487c640039fe5cb58bdf20bb"
server_key_version: "encryptionkey@0"
I20260926 20:53:29.078030 18882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:29.092980 18882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:29.093838 18882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:29.094132 18882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:29.095106 18882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:29.096516 18882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:29.096688 18882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:29.096827 18882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:29.096925 18882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:29.130096 18882 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:37271
I20260926 20:53:29.130308 19008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:37271 every 8 connection(s)
I20260926 20:53:29.131486 18882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:53:29.139021 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 18882
I20260926 20:53:29.139194 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:53:29.139698 9190 external_mini_cluster.cc:1428] Setting key 81c7d236147fd29668db9258e804004b
I20260926 20:53:29.144315 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:29.134009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:51277 (local address 127.8.249.190:44575)
0926 20:53:29.134217 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:29.134224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:29.135633 (+ 1409us) server_negotiation.cc:407] Connection header received
0926 20:53:29.137246 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:29.137250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:29.137315 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:29.137404 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:29.138521 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:29.139641 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:29.141434 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:29.142031 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:29.142181 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:29.143655 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:29.143660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:29.143663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:29.143680 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:29.143711 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:29.143723 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:29.143793 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:29.144010 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 20:53:29.144080 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:53:29.145370 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:29.134569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:44575 (local address 127.8.249.131:51277)
0926 20:53:29.135483 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:29.135518 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:29.137018 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:29.137721 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:29.137729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:29.137813 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:29.138382 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:29.138398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:29.139821 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:29.139830 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:29.141287 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:29.141298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:29.142229 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:29.142233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:29.142457 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:29.143339 (+ 882us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:53:29.143342 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:29.143383 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:53:29.143460 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:29.143816 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:29.143824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:29.143831 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:53:29.143992 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 20:53:29.144294 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":313,"threads_started":1}
I20260926 20:53:29.146763 19009 heartbeater.cc:347] Connected to a master server at 127.8.249.190:44575
I20260926 20:53:29.147136 19009 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:29.147969 19009 heartbeater.cc:511] Master 127.8.249.190:44575 requested a full tablet report, sending...
I20260926 20:53:29.149425 18562 ts_manager.cc:194] Registered new tserver with Master: 0a053f62b8d34c80810a523c041850ee (127.8.249.131:37271)
I20260926 20:53:29.150244 18562 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.249.131:51277
I20260926 20:53:29.155649 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:53:29.161844 18750 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:29.160166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.8.249.190:44575)
0926 20:53:29.160405 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:29.160413 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:29.160756 (+ 343us) server_negotiation.cc:407] Connection header received
0926 20:53:29.160954 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:29.160959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:29.161036 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:29.161141 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:29.161577 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47502: BlockingRecv error: recv got EOF from 127.0.0.1:47502 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:53:29.162609 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 18621
I20260926 20:53:29.170316 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:29.163602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.8.249.129:36199)
0926 20:53:29.164112 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:29.164130 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:29.164155 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:53:29.164215 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:29.164221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:29.164424 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:29.164556 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:29.164882 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:29.165846 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:29.167144 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:29.167766 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:29.167877 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:29.168884 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:29.168893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:29.168907 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:29.169497 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:29.169547 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:29.169566 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:29.169653 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:29.169793 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 20:53:29.169961 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":284,"threads_started":1}
I20260926 20:53:29.171681 18743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:29.256392 18748 heartbeater.cc:503] Master 127.8.249.190:44575 was elected leader, sending a full tablet report...
I20260926 20:53:29.258576 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 18621
I20260926 20:53:29.275393 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 18752
I20260926 20:53:29.283696 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:29.276315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.8.249.130:36287)
0926 20:53:29.276937 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:29.276961 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:29.276991 (+ 30us) server_negotiation.cc:407] Connection header received
0926 20:53:29.277058 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:29.277065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:29.277304 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:29.277443 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:29.277769 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:29.278784 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:29.280096 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:29.280717 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:29.280830 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:29.281938 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:29.281947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:29.281961 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:29.282638 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:29.282705 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:29.282725 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:29.282899 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:29.283152 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 20:53:29.283316 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":317,"threads_started":1}
I20260926 20:53:29.284942 18874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:29.366178 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 18752
I20260926 20:53:29.382326 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 18882
I20260926 20:53:29.390511 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:29.383354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.8.249.131:37271)
0926 20:53:29.383968 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:29.383975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:29.384004 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:53:29.384063 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:29.384069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:29.384273 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:29.384452 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:29.384906 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:29.385920 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:29.387271 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:29.387840 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:29.387975 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:29.389069 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:29.389077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:29.389091 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:29.389655 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:29.389704 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:29.389722 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:29.389866 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:29.390015 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 20:53:29.390157 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":343,"threads_started":1}
I20260926 20:53:29.392086 19004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:29.473392 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 18882
I20260926 20:53:29.489457 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 18531
I20260926 20:53:29.490670 18592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:29.592123 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 18531
2026-09-26T20:53:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2445 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T20:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:53:29Z Disabled control of system clock
I20260926 20:53:29.665393 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36981
--webserver_interface=127.8.249.190
--webserver_port=0
--builtin_ntp_servers=127.8.249.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:36981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:53:29.995751 19028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:29.996068 19028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:29.996214 19028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:30.003731 19028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:30.003855 19028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:30.003930 19028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:30.004024 19028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:30.004105 19028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:30.013736 19028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:36981
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:30.015137 19028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:30.017092 19028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:30.028967 19033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:30.029065 19034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:30.029018 19036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:30.032177 19028 server_base.cc:1034] running on GCE node
I20260926 20:53:30.033062 19028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:30.034545 19028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:30.035750 19028 hybrid_clock.cc:648] HybridClock initialized: now 1790456010035718 us; error 36 us; skew 500 ppm
I20260926 20:53:30.036099 19028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:53:30.038700 19028 webserver.cc:460] Webserver started at http://127.8.249.190:37707/ using document root <none> and password file <none>
I20260926 20:53:30.039410 19028 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:30.039501 19028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:30.039767 19028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:30.042567 19028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "9e2390711e78478ea590d430eec21351"
format_stamp: "Formatted at 2026-09-26 20:53:30 on dist-test-slave-2dg3"
server_key: "b322a6640445bdb0988fc2e05daa0407"
server_key_iv: "15c7f70c4137bfbf616f87dde35c3329"
server_key_version: "encryptionkey@0"
I20260926 20:53:30.043386 19028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "9e2390711e78478ea590d430eec21351"
format_stamp: "Formatted at 2026-09-26 20:53:30 on dist-test-slave-2dg3"
server_key: "b322a6640445bdb0988fc2e05daa0407"
server_key_iv: "15c7f70c4137bfbf616f87dde35c3329"
server_key_version: "encryptionkey@0"
I20260926 20:53:30.048647 19028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:53:30.052131 19042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:30.053474 19028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:53:30.053653 19028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "9e2390711e78478ea590d430eec21351"
format_stamp: "Formatted at 2026-09-26 20:53:30 on dist-test-slave-2dg3"
server_key: "b322a6640445bdb0988fc2e05daa0407"
server_key_iv: "15c7f70c4137bfbf616f87dde35c3329"
server_key_version: "encryptionkey@0"
I20260926 20:53:30.053864 19028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:30.059530 19028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:30.060276 19028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:30.060588 19028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:30.079284 19028 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:36981
I20260926 20:53:30.079340 19093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:36981 every 8 connection(s)
I20260926 20:53:30.080688 19028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:30.084610 19094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:30.088677 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 19028
I20260926 20:53:30.088900 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:53:30.089708 9190 external_mini_cluster.cc:1428] Setting key 99088c4e2e6f979ab2a5e8ca77802e2d
I20260926 20:53:30.098323 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:30.091922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.8.249.190:36981)
0926 20:53:30.093000 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:30.093018 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:30.093060 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:53:30.094624 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:30.094646 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:30.095182 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:30.095591 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:30.096681 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:30.096686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:30.096699 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:30.096711 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:30.096769 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:30.096787 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:30.097008 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:30.097149 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 20:53:30.097392 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":417,"threads_started":1}
I20260926 20:53:30.098413 19094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351: Bootstrap starting.
I20260926 20:53:30.101691 19094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:30.102893 19094 log.cc:824] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:30.105518 19094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351: No bootstrap required, opened a new log
I20260926 20:53:30.113027 19094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2390711e78478ea590d430eec21351" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36981 } }
I20260926 20:53:30.113528 19094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:30.113653 19094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2390711e78478ea590d430eec21351, State: Initialized, Role: FOLLOWER
I20260926 20:53:30.114274 19094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2390711e78478ea590d430eec21351" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36981 } }
I20260926 20:53:30.114511 19094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:53:30.114637 19094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:53:30.114835 19094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:30.116968 19094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2390711e78478ea590d430eec21351" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36981 } }
I20260926 20:53:30.117491 19094 leader_election.cc:302] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e2390711e78478ea590d430eec21351; no voters:
I20260926 20:53:30.118069 19094 leader_election.cc:288] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:53:30.118206 19099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:30.119078 19099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [term 1 LEADER]: Becoming Leader. State: Replica: 9e2390711e78478ea590d430eec21351, State: Running, Role: LEADER
I20260926 20:53:30.119902 19099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2390711e78478ea590d430eec21351" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36981 } }
I20260926 20:53:30.120541 19094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:30.122972 19100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e2390711e78478ea590d430eec21351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2390711e78478ea590d430eec21351" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36981 } } }
I20260926 20:53:30.123348 19100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:30.123129 19101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e2390711e78478ea590d430eec21351. Latest consensus state: current_term: 1 leader_uuid: "9e2390711e78478ea590d430eec21351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2390711e78478ea590d430eec21351" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36981 } } }
I20260926 20:53:30.123557 19101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351 [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:30.124733 19108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:30.130020 19108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:53:30.138386 19108 catalog_manager.cc:1347] Generated new cluster ID: 0ed4a7e30a4c4feebc9cc74281ad1b15
I20260926 20:53:30.138513 19108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:53:30.151136 19108 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:53:30.151572 19108 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:53:30.163345 19108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e2390711e78478ea590d430eec21351: Generated new TSK 0
I20260926 20:53:30.164263 19108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:53:30.170173 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36981
--builtin_ntp_servers=127.8.249.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:53:30.507270 19118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:30.507575 19118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:30.507748 19118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:30.514964 19118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:30.515070 19118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:30.515252 19118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:53:30.525830 19118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:30.527276 19118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:30.529256 19118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:30.543855 19123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:30.543730 19126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:30.543802 19124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:30.544665 19118 server_base.cc:1034] running on GCE node
I20260926 20:53:30.545251 19118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:30.546257 19118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:30.547482 19118 hybrid_clock.cc:648] HybridClock initialized: now 1790456010547415 us; error 64 us; skew 500 ppm
I20260926 20:53:30.547851 19118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:53:30.550316 19118 webserver.cc:460] Webserver started at http://127.8.249.129:32891/ using document root <none> and password file <none>
I20260926 20:53:30.551234 19118 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:30.551357 19118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:30.551703 19118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:30.556134 19118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "60872b2f4f86455cafb6287d9d85fff9"
format_stamp: "Formatted at 2026-09-26 20:53:30 on dist-test-slave-2dg3"
server_key: "5e3c0e849a94bb5572f1f59dbcef2490"
server_key_iv: "82487fbf3866bdf5b8c5467d91a739e2"
server_key_version: "encryptionkey@0"
I20260926 20:53:30.556946 19118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "60872b2f4f86455cafb6287d9d85fff9"
format_stamp: "Formatted at 2026-09-26 20:53:30 on dist-test-slave-2dg3"
server_key: "5e3c0e849a94bb5572f1f59dbcef2490"
server_key_iv: "82487fbf3866bdf5b8c5467d91a739e2"
server_key_version: "encryptionkey@0"
I20260926 20:53:30.562036 19118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 20:53:30.565536 19132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:30.567116 19118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:53:30.567283 19118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "60872b2f4f86455cafb6287d9d85fff9"
format_stamp: "Formatted at 2026-09-26 20:53:30 on dist-test-slave-2dg3"
server_key: "5e3c0e849a94bb5572f1f59dbcef2490"
server_key_iv: "82487fbf3866bdf5b8c5467d91a739e2"
server_key_version: "encryptionkey@0"
I20260926 20:53:30.567519 19118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:30.574122 19118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:30.575137 19118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:30.575443 19118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:30.576399 19118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:30.577804 19118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:30.577924 19118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:30.578061 19118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:30.578152 19118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:30.612366 19118 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:39267
I20260926 20:53:30.612478 19244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:39267 every 8 connection(s)
I20260926 20:53:30.613703 19118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:53:30.616544 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 19118
I20260926 20:53:30.616751 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:53:30.617559 9190 external_mini_cluster.cc:1428] Setting key 741624aeb0be917f58dbdfb796c50eba
I20260926 20:53:30.620400 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36981
--builtin_ntp_servers=127.8.249.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 20:53:30.622941 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:30.616243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:39409 (local address 127.8.249.190:36981)
0926 20:53:30.616708 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:30.616714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:30.618315 (+ 1601us) server_negotiation.cc:407] Connection header received
0926 20:53:30.620446 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:30.620452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:30.620514 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:30.620619 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:30.622210 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:30.622214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:30.622217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:30.622232 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:30.622287 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:30.622298 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:30.622386 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:30.622615 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 20:53:30.622662 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":228,"threads_started":1}
I20260926 20:53:30.624305 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:30.617040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36981 (local address 127.8.249.129:39409)
0926 20:53:30.618103 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:30.618145 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:30.620155 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:30.620938 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:30.620953 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:30.621121 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:30.621854 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:53:30.621860 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:30.621916 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:53:30.622038 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:30.622395 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:30.622408 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:30.622419 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 20:53:30.622707 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 20:53:30.623002 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":326,"threads_started":1}
I20260926 20:53:30.625902 19245 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36981
I20260926 20:53:30.626217 19245 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:30.626942 19245 heartbeater.cc:511] Master 127.8.249.190:36981 requested a full tablet report, sending...
I20260926 20:53:30.629305 19059 ts_manager.cc:194] Registered new tserver with Master: 60872b2f4f86455cafb6287d9d85fff9 (127.8.249.129:39267)
W20260926 20:53:30.952796 19249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:30.953097 19249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:30.953188 19249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:30.960863 19249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:30.960986 19249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:30.961099 19249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:53:30.970423 19249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:30.971854 19249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:30.973732 19249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:30.987397 19255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:30.987422 19254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:30.987454 19257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:30.988262 19249 server_base.cc:1034] running on GCE node
I20260926 20:53:30.988929 19249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:30.989900 19249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:30.991101 19249 hybrid_clock.cc:648] HybridClock initialized: now 1790456010991053 us; error 37 us; skew 500 ppm
I20260926 20:53:30.991410 19249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:53:30.993666 19249 webserver.cc:460] Webserver started at http://127.8.249.130:32843/ using document root <none> and password file <none>
I20260926 20:53:30.994385 19249 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:30.994475 19249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:30.994771 19249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:30.997665 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "b53fae190f8143b4a1da21462687dc7d"
format_stamp: "Formatted at 2026-09-26 20:53:30 on dist-test-slave-2dg3"
server_key: "d48c384e44bd1017d1e44a7fa1b2ce54"
server_key_iv: "359dad96d234afda0148b3f90d7229cf"
server_key_version: "encryptionkey@0"
I20260926 20:53:30.998440 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "b53fae190f8143b4a1da21462687dc7d"
format_stamp: "Formatted at 2026-09-26 20:53:30 on dist-test-slave-2dg3"
server_key: "d48c384e44bd1017d1e44a7fa1b2ce54"
server_key_iv: "359dad96d234afda0148b3f90d7229cf"
server_key_version: "encryptionkey@0"
I20260926 20:53:31.003769 19249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 20:53:31.007344 19263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:31.008806 19249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:53:31.009022 19249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "b53fae190f8143b4a1da21462687dc7d"
format_stamp: "Formatted at 2026-09-26 20:53:30 on dist-test-slave-2dg3"
server_key: "d48c384e44bd1017d1e44a7fa1b2ce54"
server_key_iv: "359dad96d234afda0148b3f90d7229cf"
server_key_version: "encryptionkey@0"
I20260926 20:53:31.009265 19249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:31.015079 19249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:31.015904 19249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:31.016189 19249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:31.017089 19249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:31.018499 19249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:31.018586 19249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:31.018760 19249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:31.018857 19249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 20:53:31.051748 19249 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:35505
I20260926 20:53:31.051795 19375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:35505 every 8 connection(s)
I20260926 20:53:31.053175 19249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:53:31.060070 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 19249
I20260926 20:53:31.060273 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:53:31.060843 9190 external_mini_cluster.cc:1428] Setting key fea612646e973a3dfbce60558b98e47e
I20260926 20:53:31.062006 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:31.055954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:56489 (local address 127.8.249.190:36981)
0926 20:53:31.056129 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:31.056134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:31.057699 (+ 1565us) server_negotiation.cc:407] Connection header received
0926 20:53:31.059576 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:31.059581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:31.059649 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:31.059747 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:31.061262 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:31.061266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:31.061270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:31.061285 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:31.061316 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:31.061328 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:31.061453 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:31.061644 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 20:53:31.061712 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:53:31.063226 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:31.056493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36981 (local address 127.8.249.130:56489)
0926 20:53:31.057519 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:31.057571 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:31.059310 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:31.060028 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:31.060042 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:31.060180 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:31.060911 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:53:31.060915 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:31.060972 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:53:31.061086 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:31.061433 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:31.061443 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:31.061457 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 20:53:31.061705 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 20:53:31.061995 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":291,"threads_started":1}
I20260926 20:53:31.063920 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36981
--builtin_ntp_servers=127.8.249.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 20:53:31.064797 19376 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36981
I20260926 20:53:31.065121 19376 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:31.065831 19376 heartbeater.cc:511] Master 127.8.249.190:36981 requested a full tablet report, sending...
I20260926 20:53:31.067461 19059 ts_manager.cc:194] Registered new tserver with Master: b53fae190f8143b4a1da21462687dc7d (127.8.249.130:35505)
W20260926 20:53:31.395098 19379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:31.395378 19379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:31.395464 19379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:31.402732 19379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:31.402856 19379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:31.402971 19379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:53:31.411988 19379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:31.413390 19379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:31.415287 19379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:31.427960 19384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:31.428076 19387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:31.428483 19385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:31.428908 19379 server_base.cc:1034] running on GCE node
I20260926 20:53:31.429481 19379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:31.430383 19379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:31.431581 19379 hybrid_clock.cc:648] HybridClock initialized: now 1790456011431525 us; error 49 us; skew 500 ppm
I20260926 20:53:31.431931 19379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:53:31.434265 19379 webserver.cc:460] Webserver started at http://127.8.249.131:42529/ using document root <none> and password file <none>
I20260926 20:53:31.435041 19379 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:31.435171 19379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:31.435499 19379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:31.438431 19379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "b43386937e3c4cfebcc88b6562c9dad1"
format_stamp: "Formatted at 2026-09-26 20:53:31 on dist-test-slave-2dg3"
server_key: "2bb2d87b7037a6beb61ae6fd9792d695"
server_key_iv: "44b712fc0861520506812cd7bc6c99e7"
server_key_version: "encryptionkey@0"
I20260926 20:53:31.439276 19379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "b43386937e3c4cfebcc88b6562c9dad1"
format_stamp: "Formatted at 2026-09-26 20:53:31 on dist-test-slave-2dg3"
server_key: "2bb2d87b7037a6beb61ae6fd9792d695"
server_key_iv: "44b712fc0861520506812cd7bc6c99e7"
server_key_version: "encryptionkey@0"
I20260926 20:53:31.444458 19379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 20:53:31.448130 19393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:31.449684 19379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:53:31.449895 19379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "b43386937e3c4cfebcc88b6562c9dad1"
format_stamp: "Formatted at 2026-09-26 20:53:31 on dist-test-slave-2dg3"
server_key: "2bb2d87b7037a6beb61ae6fd9792d695"
server_key_iv: "44b712fc0861520506812cd7bc6c99e7"
server_key_version: "encryptionkey@0"
I20260926 20:53:31.450142 19379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:31.456143 19379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:31.456992 19379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:31.457284 19379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:31.458379 19379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:31.459853 19379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:31.459976 19379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:31.460107 19379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:31.460206 19379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:31.493078 19379 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:39745
I20260926 20:53:31.493132 19505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:39745 every 8 connection(s)
I20260926 20:53:31.494381 19379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:53:31.498920 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 19379
I20260926 20:53:31.499099 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:53:31.499652 9190 external_mini_cluster.cc:1428] Setting key 0198f2515a1d8c949c30ccd7bdb8fcbf
I20260926 20:53:31.503259 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:31.496909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:57337 (local address 127.8.249.190:36981)
0926 20:53:31.497079 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:31.497083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:31.498477 (+ 1394us) server_negotiation.cc:407] Connection header received
0926 20:53:31.500707 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:31.500710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:31.500794 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:31.500889 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:31.502360 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:31.502381 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:31.502386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:31.502404 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:31.502445 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:31.502464 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:31.502586 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:31.502915 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 20:53:31.502981 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:53:31.504122 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:31.497453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36981 (local address 127.8.249.131:57337)
0926 20:53:31.498340 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:31.498374 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:31.500432 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:31.501113 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:31.501121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:31.501218 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:31.502014 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:53:31.502017 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:31.502053 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:53:31.502172 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:31.502640 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:31.502651 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:31.502659 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:53:31.502989 (+ 330us) client_negotiation.cc:240] Negotiation successful
0926 20:53:31.503228 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":286,"threads_started":1}
I20260926 20:53:31.505249 19506 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36981
I20260926 20:53:31.505493 19506 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:31.505996 19506 heartbeater.cc:511] Master 127.8.249.190:36981 requested a full tablet report, sending...
I20260926 20:53:31.507331 19059 ts_manager.cc:194] Registered new tserver with Master: b43386937e3c4cfebcc88b6562c9dad1 (127.8.249.131:39745)
I20260926 20:53:31.515283 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:53:31.521476 19247 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:53:31.519856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.8.249.190:36981)
0926 20:53:31.520045 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:31.520054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:31.520508 (+ 454us) server_negotiation.cc:407] Connection header received
0926 20:53:31.520712 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:31.520719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:31.520779 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:31.520880 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:31.521195 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43112: BlockingRecv error: recv got EOF from 127.0.0.1:43112 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:53:31.522969 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 19118
I20260926 20:53:31.526289 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:31.523856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.8.249.129:39267)
0926 20:53:31.524380 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:31.524387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:31.524413 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:53:31.524468 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:31.524474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:31.524705 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:31.524830 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:31.525722 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:31.525728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:31.525739 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:31.525758 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:31.525797 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:31.525812 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:31.525906 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:31.525987 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:53:31.526060 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":254,"threads_started":1}
I20260926 20:53:31.527547 19240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:31.606245 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 19118
I20260926 20:53:31.622485 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 19249
I20260926 20:53:31.626035 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:31.623473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.8.249.130:35505)
0926 20:53:31.623992 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:31.623999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:31.624025 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:53:31.624086 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:31.624092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:31.624310 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:31.624433 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:31.625375 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:31.625381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:31.625394 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:31.625411 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:31.625453 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:31.625467 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:31.625567 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:31.625624 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 20:53:31.625731 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":266,"threads_started":1}
I20260926 20:53:31.627405 19371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:31.708613 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 19249
I20260926 20:53:31.725315 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 19379
I20260926 20:53:31.729182 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:31.726396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.8.249.131:39745)
0926 20:53:31.727055 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:31.727062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:31.727101 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:53:31.727163 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:31.727170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:31.727400 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:31.727526 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:31.728425 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:31.728431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:31.728444 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:53:31.728458 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:31.728496 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:53:31.728509 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:31.728639 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:31.728720 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:53:31.728828 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":304,"threads_started":1}
I20260926 20:53:31.730484 19501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:31.816553 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 19379
I20260926 20:53:31.832190 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 19028
I20260926 20:53:31.833406 19089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:31.934157 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 19028
2026-09-26T20:53:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2341 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:53:32 dist-test-slave-2dg3 krb5kdc[19524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:53:32 dist-test-slave-2dg3 krb5kdc[19524](info): set up 2 sockets
Sep 26 20:53:32 dist-test-slave-2dg3 krb5kdc[19524](info): commencing operation
krb5kdc: starting...
W20260926 20:53:33.027853 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:53:33 dist-test-slave-2dg3 krb5kdc[19524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:53:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:53:33.276468 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36643
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:36643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:33.613055 19537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:33.613386 19537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:33.613543 19537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:33.620975 19537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:53:33.621106 19537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:33.621207 19537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:33.621321 19537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:53:33.621404 19537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:53:33.631552 19537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:36643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:33.633001 19537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:33.634444 19537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:33.647375 19542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:33.647348 19543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:33.647320 19545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:33.648736 19537 server_base.cc:1034] running on GCE node
I20260926 20:53:33.649714 19537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:33.651315 19537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:33.652596 19537 hybrid_clock.cc:648] HybridClock initialized: now 1790456013652543 us; error 41 us; skew 500 ppm
I20260926 20:53:33.652981 19537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:53:33 dist-test-slave-2dg3 krb5kdc[19524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456013, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:33.661504 19537 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:53:33.663157 19537 webserver.cc:460] Webserver started at http://127.8.249.190:38769/ using document root <none> and password file <none>
I20260926 20:53:33.663861 19537 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:33.663966 19537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:33.664233 19537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:33.667390 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "53beb29f7275438689646b419735cf2e"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-2dg3"
server_key: "3ecb1b2e23faa70f05251af7ea977b70"
server_key_iv: "809787d9079a25aa66fd665d311a7aaa"
server_key_version: "encryptionkey@0"
I20260926 20:53:33.668133 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "53beb29f7275438689646b419735cf2e"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-2dg3"
server_key: "3ecb1b2e23faa70f05251af7ea977b70"
server_key_iv: "809787d9079a25aa66fd665d311a7aaa"
server_key_version: "encryptionkey@0"
I20260926 20:53:33.673724 19537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:53:33.677726 19552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:33.679620 19537 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:53:33.679795 19537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "53beb29f7275438689646b419735cf2e"
format_stamp: "Formatted at 2026-09-26 20:53:33 on dist-test-slave-2dg3"
server_key: "3ecb1b2e23faa70f05251af7ea977b70"
server_key_iv: "809787d9079a25aa66fd665d311a7aaa"
server_key_version: "encryptionkey@0"
I20260926 20:53:33.680053 19537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:33.695665 19537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:33.700095 19537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:33.700400 19537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:33.719642 19537 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:36643
I20260926 20:53:33.719676 19603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:36643 every 8 connection(s)
I20260926 20:53:33.721138 19537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:53:33.724918 19604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:33.731361 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 19537
I20260926 20:53:33.731657 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:53:33.732313 9190 external_mini_cluster.cc:1428] Setting key 14e1310409d08d252f0f30ddc0bd515a
I20260926 20:53:33.737406 19604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e: Bootstrap starting.
I20260926 20:53:33.741395 19604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:33.742710 19604 log.cc:824] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:33.746182 19604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e: No bootstrap required, opened a new log
Sep 26 20:53:33 dist-test-slave-2dg3 krb5kdc[19524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:53:33.755844 19604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53beb29f7275438689646b419735cf2e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36643 } }
I20260926 20:53:33.756366 19604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:33.756476 19604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53beb29f7275438689646b419735cf2e, State: Initialized, Role: FOLLOWER
I20260926 20:53:33.757189 19604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53beb29f7275438689646b419735cf2e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36643 } }
I20260926 20:53:33.757474 19604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:53:33.757609 19604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:53:33.757807 19604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:33.759586 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:33.734686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.8.249.190:36643)
0926 20:53:33.735580 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:33.735597 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:33.735649 (+ 52us) server_negotiation.cc:407] Connection header received
0926 20:53:33.737975 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:33.738011 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:33.738546 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:33.739033 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:33.740401 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.742412 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.744391 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:33.745403 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:33.745596 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:33.749877 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:33.749892 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:33.749913 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:33.749969 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:33.754414 (+ 4445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:33.755287 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:33.755296 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:33.755305 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:33.755515 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:33.755936 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:33.755941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:33.755944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:33.756694 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:33.756996 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:33.757445 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 20:53:33.758118 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":353,"threads_started":1}
I20260926 20:53:33.761142 19604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53beb29f7275438689646b419735cf2e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36643 } }
I20260926 20:53:33.761663 19604 leader_election.cc:302] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53beb29f7275438689646b419735cf2e; no voters:
I20260926 20:53:33.762281 19604 leader_election.cc:288] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:53:33.762435 19609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:33.764008 19609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [term 1 LEADER]: Becoming Leader. State: Replica: 53beb29f7275438689646b419735cf2e, State: Running, Role: LEADER
I20260926 20:53:33.764698 19604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:53:33.764591 19609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53beb29f7275438689646b419735cf2e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36643 } }
I20260926 20:53:33.767628 19611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53beb29f7275438689646b419735cf2e. Latest consensus state: current_term: 1 leader_uuid: "53beb29f7275438689646b419735cf2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53beb29f7275438689646b419735cf2e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36643 } } }
I20260926 20:53:33.768371 19611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:33.769013 19610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53beb29f7275438689646b419735cf2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53beb29f7275438689646b419735cf2e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36643 } } }
I20260926 20:53:33.769394 19610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e [sys.catalog]: This master's current role is: LEADER
I20260926 20:53:33.777040 19619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:53:33.784162 19619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:53:33.795375 19619 catalog_manager.cc:1347] Generated new cluster ID: 29581c9447e34d1faf6dae0d854f4813
I20260926 20:53:33.795507 19619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:53:33.818327 19619 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:53:33.820295 19619 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:53:33.827270 19619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e: Generated new TSK 0
I20260926 20:53:33.828329 19619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:53:33.887388 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36643
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:34.228703 19630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:34.229144 19630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:34.229363 19630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:34.239890 19630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:34.240025 19630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:34.240223 19630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:53:34.250300 19630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:34.251688 19630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:34.253047 19630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:34.267422 19638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:34.267458 19635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:34.267437 19636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:34.268232 19630 server_base.cc:1034] running on GCE node
I20260926 20:53:34.268863 19630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:34.269897 19630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:34.271096 19630 hybrid_clock.cc:648] HybridClock initialized: now 1790456014271032 us; error 54 us; skew 500 ppm
I20260926 20:53:34.271430 19630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:53:34 dist-test-slave-2dg3 krb5kdc[19524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456014, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:34.274570 19630 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:53:34.276015 19630 webserver.cc:460] Webserver started at http://127.8.249.129:33065/ using document root <none> and password file <none>
I20260926 20:53:34.276679 19630 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:34.276779 19630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:34.277031 19630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:34.279973 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "a51dc912c06742e1894e007e40f0ac77"
format_stamp: "Formatted at 2026-09-26 20:53:34 on dist-test-slave-2dg3"
server_key: "02144719cee93fc5cf7f459fc66f6314"
server_key_iv: "7442c5a5ee62a245536a92ee8c83af84"
server_key_version: "encryptionkey@0"
I20260926 20:53:34.280774 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "a51dc912c06742e1894e007e40f0ac77"
format_stamp: "Formatted at 2026-09-26 20:53:34 on dist-test-slave-2dg3"
server_key: "02144719cee93fc5cf7f459fc66f6314"
server_key_iv: "7442c5a5ee62a245536a92ee8c83af84"
server_key_version: "encryptionkey@0"
I20260926 20:53:34.286019 19630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 20:53:34.289423 19645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:34.290746 19630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:53:34.290947 19630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "a51dc912c06742e1894e007e40f0ac77"
format_stamp: "Formatted at 2026-09-26 20:53:34 on dist-test-slave-2dg3"
server_key: "02144719cee93fc5cf7f459fc66f6314"
server_key_iv: "7442c5a5ee62a245536a92ee8c83af84"
server_key_version: "encryptionkey@0"
I20260926 20:53:34.291146 19630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:34.309795 19630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:34.314239 19630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:34.314543 19630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:34.315546 19630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:34.316852 19630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:34.316923 19630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:34.317086 19630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:34.317143 19630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:34.348764 19630 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:45041
I20260926 20:53:34.348800 19757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:45041 every 8 connection(s)
I20260926 20:53:34.350056 19630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:53:34.351857 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 19630
I20260926 20:53:34.352095 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:53:34.352797 9190 external_mini_cluster.cc:1428] Setting key 283e6d33e4c315efe5556fb5ec45493e
Sep 26 20:53:34 dist-test-slave-2dg3 krb5kdc[19524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456014, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:53:34.371258 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:34.352956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:56535 (local address 127.8.249.190:36643)
0926 20:53:34.353456 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:34.353462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:34.354637 (+ 1175us) server_negotiation.cc:407] Connection header received
0926 20:53:34.356510 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:34.356516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:34.356608 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:34.356727 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:34.358260 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:34.359077 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:34.360685 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:34.361086 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:34.361243 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:34.365280 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:34.365285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:34.365289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:34.365324 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:34.368051 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:34.368888 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:34.368897 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:34.368901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:34.369022 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:34.369470 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:34.369478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:34.369482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:34.369748 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:34.369890 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:34.370621 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 20:53:34.370882 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260926 20:53:34.372367 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:34.353625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36643 (local address 127.8.249.129:56535)
0926 20:53:34.354490 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:34.354536 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:34.356238 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:34.356983 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:34.356992 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:34.357504 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:34.358081 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:34.358098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:34.359239 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:34.359245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:34.360508 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:34.360520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:34.361357 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:34.361364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:34.361606 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:34.362344 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:34.362371 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:34.365017 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:34.368241 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:34.368252 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:34.368273 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:34.368762 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:34.369143 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:34.369148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:34.369152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:34.369362 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:34.369880 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:34.369889 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:34.370260 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 20:53:34.370539 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 20:53:34.371156 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":354,"threads_started":1}
I20260926 20:53:34.374078 19758 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36643
I20260926 20:53:34.374441 19758 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
I20260926 20:53:34.375531 19758 heartbeater.cc:511] Master 127.8.249.190:36643 requested a full tablet report, sending...
I20260926 20:53:34.377974 19569 ts_manager.cc:194] Registered new tserver with Master: a51dc912c06742e1894e007e40f0ac77 (127.8.249.129:45041)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
I20260926 20:53:34.380275 19569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:56535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:53:34.410712 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36643
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:34.744304 19764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:34.744581 19764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:34.744675 19764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:34.751952 19764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:34.752125 19764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:34.752256 19764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:53:34.762135 19764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:34.763540 19764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:34.764844 19764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:34.778664 19772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:34.778854 19769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:34.778838 19770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:34.779956 19764 server_base.cc:1034] running on GCE node
I20260926 20:53:34.780820 19764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:34.782015 19764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:34.783293 19764 hybrid_clock.cc:648] HybridClock initialized: now 1790456014783218 us; error 73 us; skew 500 ppm
I20260926 20:53:34.783684 19764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:53:34 dist-test-slave-2dg3 krb5kdc[19524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456014, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:34.787142 19764 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:53:34.788751 19764 webserver.cc:460] Webserver started at http://127.8.249.130:46529/ using document root <none> and password file <none>
I20260926 20:53:34.789494 19764 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:34.789652 19764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:34.789963 19764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:34.793028 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "3f64949b71564d8eb7b5ba8a475ae3c3"
format_stamp: "Formatted at 2026-09-26 20:53:34 on dist-test-slave-2dg3"
server_key: "73557f082c701537bcfd7f89aedaebf5"
server_key_iv: "8641e475b74fa9e086677f1a107dc078"
server_key_version: "encryptionkey@0"
I20260926 20:53:34.793825 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "3f64949b71564d8eb7b5ba8a475ae3c3"
format_stamp: "Formatted at 2026-09-26 20:53:34 on dist-test-slave-2dg3"
server_key: "73557f082c701537bcfd7f89aedaebf5"
server_key_iv: "8641e475b74fa9e086677f1a107dc078"
server_key_version: "encryptionkey@0"
I20260926 20:53:34.799351 19764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:53:34.803015 19779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:34.804562 19764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:53:34.804780 19764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "3f64949b71564d8eb7b5ba8a475ae3c3"
format_stamp: "Formatted at 2026-09-26 20:53:34 on dist-test-slave-2dg3"
server_key: "73557f082c701537bcfd7f89aedaebf5"
server_key_iv: "8641e475b74fa9e086677f1a107dc078"
server_key_version: "encryptionkey@0"
I20260926 20:53:34.804994 19764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:34.842904 19764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:34.846958 19764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:34.847251 19764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:34.848169 19764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:34.849450 19764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:34.849565 19764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:34.849704 19764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:34.849777 19764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:34.882674 19764 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:40123
I20260926 20:53:34.882685 19891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:40123 every 8 connection(s)
I20260926 20:53:34.884105 19764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:53:34.884629 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 19764
I20260926 20:53:34.884956 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:53:34.885514 9190 external_mini_cluster.cc:1428] Setting key 597f5522065a3f1d96d755a384f0c1df
Sep 26 20:53:34 dist-test-slave-2dg3 krb5kdc[19524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456014, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:53:34.910573 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:34.886876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:48301 (local address 127.8.249.190:36643)
0926 20:53:34.891179 (+ 4303us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:34.891183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:34.891204 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:53:34.891932 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:34.891937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:34.892018 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:34.892118 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:34.894498 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:34.895432 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:34.897536 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:34.898343 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:34.898517 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:34.902449 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:34.902459 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:34.902468 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:34.902523 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:34.906495 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:34.907432 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:34.907441 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:34.907448 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:34.907625 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:34.908091 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:34.908099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:34.908106 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:34.908608 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:34.908814 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:34.909588 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 20:53:34.909871 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4190,"thread_start_us":4092,"threads_started":1}
I20260926 20:53:34.911371 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:34.887788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36643 (local address 127.8.249.130:48301)
0926 20:53:34.888782 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:34.888835 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:34.891562 (+ 2727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:34.892436 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:34.892451 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:34.893317 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:34.894237 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:34.894268 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:34.895626 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:34.895634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:34.897353 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:34.897366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:34.898574 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:34.898579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:34.898864 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:34.899616 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:34.899639 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:34.902163 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:34.906691 (+ 4528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:34.906699 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:34.906715 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:34.907293 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:34.907746 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:34.907750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:34.907752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:34.907943 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:34.908753 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:34.908759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:34.909156 (+ 397us) client_negotiation.cc:769] Sending connection context
0926 20:53:34.909445 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 20:53:34.910110 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":387,"threads_started":1}
I20260926 20:53:34.912856 19892 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36643
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
I20260926 20:53:34.913197 19892 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:34.913983 19892 heartbeater.cc:511] Master 127.8.249.190:36643 requested a full tablet report, sending...
I20260926 20:53:34.915493 19569 ts_manager.cc:194] Registered new tserver with Master: 3f64949b71564d8eb7b5ba8a475ae3c3 (127.8.249.130:40123)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
I20260926 20:53:34.916507 19569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:48301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:53:34.944118 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36643
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:53:35.277302 19898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:53:35.277632 19898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:53:35.277803 19898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:53:35.285561 19898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:53:35.285738 19898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:53:35.285907 19898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:53:35.296582 19898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:53:35.297971 19898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:53:35.299412 19898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:53:35.313081 19903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:35.313170 19904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:53:35.313575 19906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:53:35.314155 19898 server_base.cc:1034] running on GCE node
I20260926 20:53:35.314884 19898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:53:35.315878 19898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:53:35.317063 19898 hybrid_clock.cc:648] HybridClock initialized: now 1790456015317019 us; error 41 us; skew 500 ppm
I20260926 20:53:35.317376 19898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:53:35 dist-test-slave-2dg3 krb5kdc[19524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:53:35.320508 19898 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:53:35.321902 19898 webserver.cc:460] Webserver started at http://127.8.249.131:35645/ using document root <none> and password file <none>
I20260926 20:53:35.322538 19898 fs_manager.cc:359] Metadata directory not provided
I20260926 20:53:35.322636 19898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:53:35.322955 19898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:53:35.325754 19898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "e17f6873c01448bba6daa3ad946a8fd9"
format_stamp: "Formatted at 2026-09-26 20:53:35 on dist-test-slave-2dg3"
server_key: "28613e416f2f7f192e843ad51b01bee7"
server_key_iv: "181fe42c593f9979e46ea1dc538e1710"
server_key_version: "encryptionkey@0"
I20260926 20:53:35.326462 19898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "e17f6873c01448bba6daa3ad946a8fd9"
format_stamp: "Formatted at 2026-09-26 20:53:35 on dist-test-slave-2dg3"
server_key: "28613e416f2f7f192e843ad51b01bee7"
server_key_iv: "181fe42c593f9979e46ea1dc538e1710"
server_key_version: "encryptionkey@0"
I20260926 20:53:35.331607 19898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 20:53:35.334913 19913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:35.336354 19898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:53:35.336620 19898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "e17f6873c01448bba6daa3ad946a8fd9"
format_stamp: "Formatted at 2026-09-26 20:53:35 on dist-test-slave-2dg3"
server_key: "28613e416f2f7f192e843ad51b01bee7"
server_key_iv: "181fe42c593f9979e46ea1dc538e1710"
server_key_version: "encryptionkey@0"
I20260926 20:53:35.336836 19898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:53:35.375742 19898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:53:35.380009 19898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:53:35.380306 19898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:53:35.381294 19898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:53:35.382561 19898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:53:35.382639 19898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:35.382725 19898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:53:35.382768 19898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:53:35.385178 19758 heartbeater.cc:503] Master 127.8.249.190:36643 was elected leader, sending a full tablet report...
I20260926 20:53:35.416210 19898 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:45571
I20260926 20:53:35.416256 20025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:45571 every 8 connection(s)
I20260926 20:53:35.417526 19898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:53:35.418498 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 19898
I20260926 20:53:35.418710 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:53:35.419296 9190 external_mini_cluster.cc:1428] Setting key 024b146b4505553304ae10ff312b94cd
Sep 26 20:53:35 dist-test-slave-2dg3 krb5kdc[19524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:53:35.437757 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.420480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:47981 (local address 127.8.249.190:36643)
0926 20:53:35.420969 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:35.420974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:35.422110 (+ 1136us) server_negotiation.cc:407] Connection header received
0926 20:53:35.423975 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:35.423980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:35.424062 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:35.424179 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:35.425616 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.426387 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.427979 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.428378 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.428504 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.431910 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:35.431914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:35.431918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:35.431948 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:35.434739 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:35.435491 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:35.435496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:35.435499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:35.435617 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:35.436024 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:35.436029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:35.436033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:35.436311 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:35.436473 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:35.437166 (+ 693us) server_negotiation.cc:299] Negotiation successful
0926 20:53:35.437382 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":234,"threads_started":1}
I20260926 20:53:35.438827 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.421224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36643 (local address 127.8.249.131:47981)
0926 20:53:35.421957 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:35.422008 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:35.423727 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:35.424405 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:35.424413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:35.424902 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:53:35.425458 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:35.425474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.426536 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.426540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:35.427841 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:35.427853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.428597 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.428605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:35.428832 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.429468 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:53:35.429490 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:53:35.431714 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:53:35.434910 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:35.434920 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:35.434938 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:35.435366 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:35.435719 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:53:35.435725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:53:35.435729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:53:35.435911 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:53:35.436430 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:53:35.436439 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:53:35.436801 (+ 362us) client_negotiation.cc:769] Sending connection context
0926 20:53:35.437059 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 20:53:35.437597 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":318,"threads_started":1}
I20260926 20:53:35.440227 20026 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36643
I20260926 20:53:35.440526 20026 heartbeater.cc:464] Registering TS with master...
I20260926 20:53:35.441244 20026 heartbeater.cc:511] Master 127.8.249.190:36643 requested a full tablet report, sending...
I20260926 20:53:35.442687 19569 ts_manager.cc:194] Registered new tserver with Master: e17f6873c01448bba6daa3ad946a8fd9 (127.8.249.131:45571)
I20260926 20:53:35.443542 19569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:47981
I20260926 20:53:35.447261 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:53:35.464360 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.451591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.8.249.190:36643)
0926 20:53:35.451821 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:35.451827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:35.452129 (+ 302us) server_negotiation.cc:407] Connection header received
0926 20:53:35.452272 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:35.452275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:35.452337 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:35.452417 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:35.453724 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.454434 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.455908 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.456350 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.456460 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.458727 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:35.458732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:35.458736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:35.458770 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:35.461523 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:35.462271 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:35.462276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:35.462278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:35.462407 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:35.462924 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:35.462928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:35.462930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:35.463281 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:35.463464 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:35.463843 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 20:53:35.464021 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:53:35.469043 19569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:53:35.475948 19569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:53:35.510568 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.498916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.8.249.129:45041)
0926 20:53:35.502325 (+ 3409us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:35.502336 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:35.503337 (+ 1001us) server_negotiation.cc:407] Connection header received
0926 20:53:35.503594 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:35.503603 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:35.503837 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:35.504005 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:35.504633 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.506175 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.508410 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.509458 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.509600 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.509826 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:35.510056 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 20:53:35.510242 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3236,"thread_start_us":3080,"threads_started":1}
I20260926 20:53:35.510622 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.502468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:45041 (local address 127.0.0.1:33390)
0926 20:53:35.503220 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:35.503235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:35.503379 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:35.504116 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:35.504121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:35.504149 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:35.504462 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:35.504470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.506344 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.506349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:35.508257 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:35.508271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.509676 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.509683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:35.509861 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.509893 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 20:53:35.510011 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:53:35.510210 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":2,"thread_start_us":501,"threads_started":1}
I20260926 20:53:35.510759 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.488209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.8.249.130:40123)
0926 20:53:35.501272 (+ 13063us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:35.501285 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:35.501319 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:53:35.501428 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:35.501436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:35.501737 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:35.501943 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:35.503019 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.504495 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.508097 (+ 3602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.509318 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.509469 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.509690 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:35.510147 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 20:53:35.510315 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12861,"thread_start_us":12730,"threads_started":1}
I20260926 20:53:35.511277 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.488055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:40123 (local address 127.0.0.1:35234)
0926 20:53:35.497894 (+ 9839us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:35.497959 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:35.498197 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:35.502245 (+ 4048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:35.502254 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:35.502294 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:35.502771 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:35.502784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.504652 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.504656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:35.507939 (+ 3283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:35.507952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.509525 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.509532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:35.509750 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.509806 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 20:53:35.509989 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 20:53:35.510927 (+ 938us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9482,"mutex_wait_us":1455,"thread_start_us":13665,"threads_started":1}
I20260926 20:53:35.513068 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.488279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:45571 (local address 127.0.0.1:35992)
0926 20:53:35.501491 (+ 13212us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:35.501509 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:35.501614 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:35.503495 (+ 1881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:35.503503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:35.503528 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:35.503891 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:35.503903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.505456 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.505461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:35.507940 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:35.507952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.512261 (+ 4309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.512268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:35.512462 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.512496 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:53:35.512571 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:53:35.512716 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13107,"mutex_wait_us":74,"thread_start_us":9084,"threads_started":1}
I20260926 20:53:35.514106 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.489049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.8.249.131:45571)
0926 20:53:35.502889 (+ 13840us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:35.502899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:35.502933 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:53:35.503026 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:35.503033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:35.503298 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:35.503498 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:35.504066 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.505297 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.509789 (+ 4492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.512012 (+ 2223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.513299 (+ 1287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.513538 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:35.513615 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 20:53:35.513787 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13643,"thread_start_us":13493,"threads_started":1}
I20260926 20:53:35.515640 19827 tablet_service.cc:1467] Processing CreateTablet for tablet c30b5adbae424066a79035484016bc75 (DEFAULT_TABLE table=test-table [id=deee3e7a2c254d8b9aefcc80f3e1e474]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:35.517194 19827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30b5adbae424066a79035484016bc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:35.517022 19961 tablet_service.cc:1467] Processing CreateTablet for tablet c30b5adbae424066a79035484016bc75 (DEFAULT_TABLE table=test-table [id=deee3e7a2c254d8b9aefcc80f3e1e474]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:35.517081 19693 tablet_service.cc:1467] Processing CreateTablet for tablet c30b5adbae424066a79035484016bc75 (DEFAULT_TABLE table=test-table [id=deee3e7a2c254d8b9aefcc80f3e1e474]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:53:35.518203 19961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30b5adbae424066a79035484016bc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:35.518438 19693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30b5adbae424066a79035484016bc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:53:35.529867 20045 tablet_bootstrap.cc:492] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9: Bootstrap starting.
I20260926 20:53:35.532225 20046 tablet_bootstrap.cc:492] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3: Bootstrap starting.
I20260926 20:53:35.534329 20045 tablet_bootstrap.cc:654] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:35.535485 20047 tablet_bootstrap.cc:492] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77: Bootstrap starting.
I20260926 20:53:35.535699 20045 log.cc:824] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:35.535823 20046 tablet_bootstrap.cc:654] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:35.536935 20046 log.cc:824] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:35.538964 20047 tablet_bootstrap.cc:654] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77: Neither blocks nor log segments found. Creating new log.
I20260926 20:53:35.539515 20045 tablet_bootstrap.cc:492] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9: No bootstrap required, opened a new log
I20260926 20:53:35.539940 20046 tablet_bootstrap.cc:492] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3: No bootstrap required, opened a new log
I20260926 20:53:35.540002 20045 ts_tablet_manager.cc:1397] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 20:53:35.540146 20047 log.cc:824] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77: Log is configured to *not* fsync() on all Append() calls
I20260926 20:53:35.540279 20046 ts_tablet_manager.cc:1397] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260926 20:53:35.542853 20047 tablet_bootstrap.cc:492] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77: No bootstrap required, opened a new log
I20260926 20:53:35.543097 20047 ts_tablet_manager.cc:1397] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 20:53:35.550202 20047 raft_consensus.cc:348] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f64949b71564d8eb7b5ba8a475ae3c3" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 40123 } } peers { permanent_uuid: "e17f6873c01448bba6daa3ad946a8fd9" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 45571 } } peers { permanent_uuid: "a51dc912c06742e1894e007e40f0ac77" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45041 } }
I20260926 20:53:35.550604 20047 raft_consensus.cc:374] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:35.550733 20047 raft_consensus.cc:729] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51dc912c06742e1894e007e40f0ac77, State: Initialized, Role: FOLLOWER
I20260926 20:53:35.551090 20046 raft_consensus.cc:348] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f64949b71564d8eb7b5ba8a475ae3c3" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 40123 } } peers { permanent_uuid: "e17f6873c01448bba6daa3ad946a8fd9" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 45571 } } peers { permanent_uuid: "a51dc912c06742e1894e007e40f0ac77" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45041 } }
I20260926 20:53:35.551090 20045 raft_consensus.cc:348] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f64949b71564d8eb7b5ba8a475ae3c3" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 40123 } } peers { permanent_uuid: "e17f6873c01448bba6daa3ad946a8fd9" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 45571 } } peers { permanent_uuid: "a51dc912c06742e1894e007e40f0ac77" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45041 } }
I20260926 20:53:35.551528 20045 raft_consensus.cc:374] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:35.551528 20046 raft_consensus.cc:374] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:53:35.551379 20047 consensus_queue.cc:260] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f64949b71564d8eb7b5ba8a475ae3c3" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 40123 } } peers { permanent_uuid: "e17f6873c01448bba6daa3ad946a8fd9" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 45571 } } peers { permanent_uuid: "a51dc912c06742e1894e007e40f0ac77" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45041 } }
I20260926 20:53:35.551631 20045 raft_consensus.cc:729] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e17f6873c01448bba6daa3ad946a8fd9, State: Initialized, Role: FOLLOWER
I20260926 20:53:35.551631 20046 raft_consensus.cc:729] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f64949b71564d8eb7b5ba8a475ae3c3, State: Initialized, Role: FOLLOWER
I20260926 20:53:35.552306 20045 consensus_queue.cc:260] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f64949b71564d8eb7b5ba8a475ae3c3" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 40123 } } peers { permanent_uuid: "e17f6873c01448bba6daa3ad946a8fd9" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 45571 } } peers { permanent_uuid: "a51dc912c06742e1894e007e40f0ac77" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45041 } }
I20260926 20:53:35.552306 20046 consensus_queue.cc:260] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f64949b71564d8eb7b5ba8a475ae3c3" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 40123 } } peers { permanent_uuid: "e17f6873c01448bba6daa3ad946a8fd9" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 45571 } } peers { permanent_uuid: "a51dc912c06742e1894e007e40f0ac77" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45041 } }
I20260926 20:53:35.556730 19892 heartbeater.cc:503] Master 127.8.249.190:36643 was elected leader, sending a full tablet report...
I20260926 20:53:35.558091 20046 ts_tablet_manager.cc:1428] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260926 20:53:35.559952 20047 ts_tablet_manager.cc:1428] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260926 20:53:35.560300 20045 ts_tablet_manager.cc:1428] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9: Time spent starting tablet: real 0.020s user 0.014s sys 0.005s
I20260926 20:53:35.560657 20026 heartbeater.cc:503] Master 127.8.249.190:36643 was elected leader, sending a full tablet report...
W20260926 20:53:35.602471 19759 tablet.cc:2369] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:53:35.636552 19893 tablet.cc:2369] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:53:35.669355 20027 tablet.cc:2369] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:53:35.729199 20053 raft_consensus.cc:482] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:53:35.729524 20053 raft_consensus.cc:504] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f64949b71564d8eb7b5ba8a475ae3c3" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 40123 } } peers { permanent_uuid: "e17f6873c01448bba6daa3ad946a8fd9" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 45571 } } peers { permanent_uuid: "a51dc912c06742e1894e007e40f0ac77" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45041 } }
I20260926 20:53:35.731278 20053 leader_election.cc:288] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f64949b71564d8eb7b5ba8a475ae3c3 (127.8.249.130:40123), a51dc912c06742e1894e007e40f0ac77 (127.8.249.129:45041)
I20260926 20:53:35.737284 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.731513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:40123 (local address 127.8.249.131:37871)
0926 20:53:35.731910 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:35.731930 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:35.732046 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:35.732848 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:35.732851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:35.732870 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:35.733017 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:35.733024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.734027 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.734034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:35.735428 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:35.735440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.736531 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.736537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:35.736715 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.736738 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:53:35.736842 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:53:35.736992 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247}
I20260926 20:53:35.737260 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.732303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:37871 (local address 127.8.249.130:40123)
0926 20:53:35.732502 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:35.732507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:35.732526 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:53:35.732601 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:35.732605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:35.732700 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:35.732818 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:35.733164 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.733891 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.735596 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.736375 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.736538 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.736612 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:35.736869 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 20:53:35.736995 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:53:35.737617 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.731798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:48565 (local address 127.8.249.129:45041)
0926 20:53:35.732064 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:35.732073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:35.732665 (+ 592us) server_negotiation.cc:407] Connection header received
0926 20:53:35.732827 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:35.732832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:35.732903 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:35.732993 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:53:35.733499 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.734181 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.735939 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.736711 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.736897 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.736969 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:35.737241 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 20:53:35.737368 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 20:53:35.737668 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.731692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:45041 (local address 127.8.249.131:48565)
0926 20:53:35.732552 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 20:53:35.732568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:53:35.732688 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:53:35.733010 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:53:35.733015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:53:35.733051 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:53:35.733337 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:35.733345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.734328 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.734334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:35.735771 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:53:35.735784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.736915 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.736921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:53:35.737074 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.737097 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:53:35.737212 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:53:35.737338 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":587,"threads_started":1}
I20260926 20:53:35.738407 19847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30b5adbae424066a79035484016bc75" candidate_uuid: "e17f6873c01448bba6daa3ad946a8fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f64949b71564d8eb7b5ba8a475ae3c3" is_pre_election: true
I20260926 20:53:35.738689 19713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30b5adbae424066a79035484016bc75" candidate_uuid: "e17f6873c01448bba6daa3ad946a8fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51dc912c06742e1894e007e40f0ac77" is_pre_election: true
I20260926 20:53:35.738945 19847 raft_consensus.cc:2454] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e17f6873c01448bba6daa3ad946a8fd9 in term 0.
I20260926 20:53:35.739238 19713 raft_consensus.cc:2454] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e17f6873c01448bba6daa3ad946a8fd9 in term 0.
I20260926 20:53:35.739636 19917 leader_election.cc:302] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f64949b71564d8eb7b5ba8a475ae3c3, e17f6873c01448bba6daa3ad946a8fd9; no voters:
I20260926 20:53:35.740099 20053 raft_consensus.cc:2781] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:53:35.740249 20053 raft_consensus.cc:482] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:53:35.740367 20053 raft_consensus.cc:3037] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:35.742616 20053 raft_consensus.cc:504] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f64949b71564d8eb7b5ba8a475ae3c3" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 40123 } } peers { permanent_uuid: "e17f6873c01448bba6daa3ad946a8fd9" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 45571 } } peers { permanent_uuid: "a51dc912c06742e1894e007e40f0ac77" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45041 } }
I20260926 20:53:35.743251 20053 leader_election.cc:288] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [CANDIDATE]: Term 1 election: Requested vote from peers 3f64949b71564d8eb7b5ba8a475ae3c3 (127.8.249.130:40123), a51dc912c06742e1894e007e40f0ac77 (127.8.249.129:45041)
I20260926 20:53:35.743796 19847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30b5adbae424066a79035484016bc75" candidate_uuid: "e17f6873c01448bba6daa3ad946a8fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f64949b71564d8eb7b5ba8a475ae3c3"
I20260926 20:53:35.743795 19713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30b5adbae424066a79035484016bc75" candidate_uuid: "e17f6873c01448bba6daa3ad946a8fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51dc912c06742e1894e007e40f0ac77"
I20260926 20:53:35.744005 19713 raft_consensus.cc:3037] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:35.744005 19847 raft_consensus.cc:3037] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:53:35.747198 19713 raft_consensus.cc:2454] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e17f6873c01448bba6daa3ad946a8fd9 in term 1.
I20260926 20:53:35.747196 19847 raft_consensus.cc:2454] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e17f6873c01448bba6daa3ad946a8fd9 in term 1.
I20260926 20:53:35.747651 19917 leader_election.cc:302] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f64949b71564d8eb7b5ba8a475ae3c3, e17f6873c01448bba6daa3ad946a8fd9; no voters:
I20260926 20:53:35.747956 20053 raft_consensus.cc:2781] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:53:35.748432 20053 raft_consensus.cc:686] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [term 1 LEADER]: Becoming Leader. State: Replica: e17f6873c01448bba6daa3ad946a8fd9, State: Running, Role: LEADER
I20260926 20:53:35.748950 20053 consensus_queue.cc:237] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f64949b71564d8eb7b5ba8a475ae3c3" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 40123 } } peers { permanent_uuid: "e17f6873c01448bba6daa3ad946a8fd9" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 45571 } } peers { permanent_uuid: "a51dc912c06742e1894e007e40f0ac77" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45041 } }
I20260926 20:53:35.753839 19569 catalog_manager.cc:5495] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e17f6873c01448bba6daa3ad946a8fd9 (127.8.249.131). New cstate: current_term: 1 leader_uuid: "e17f6873c01448bba6daa3ad946a8fd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f64949b71564d8eb7b5ba8a475ae3c3" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 40123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e17f6873c01448bba6daa3ad946a8fd9" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 45571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a51dc912c06742e1894e007e40f0ac77" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45041 } health_report { overall_health: UNKNOWN } } }
I20260926 20:53:35.796379 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.791313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.8.249.131:45571)
0926 20:53:35.791489 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:35.791494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:35.791634 (+ 140us) server_negotiation.cc:407] Connection header received
0926 20:53:35.791790 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:35.791794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:35.791876 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:35.791995 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:53:35.792301 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.793007 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.794389 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.794837 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.794930 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.795354 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:53:35.795899 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:53:35.796004 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:35.796122 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 20:53:35.796185 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:53:35.804461 19847 raft_consensus.cc:1261] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3 [term 1 FOLLOWER]: Refusing update from remote peer e17f6873c01448bba6daa3ad946a8fd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:35.804469 19713 raft_consensus.cc:1261] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77 [term 1 FOLLOWER]: Refusing update from remote peer e17f6873c01448bba6daa3ad946a8fd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:53:35.805401 20055 consensus_queue.cc:1035] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51dc912c06742e1894e007e40f0ac77" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:35.805716 20053 consensus_queue.cc:1035] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f64949b71564d8eb7b5ba8a475ae3c3" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 40123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:53:35.832820 19569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:35.833117 19569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:53:35.836704 19569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53beb29f7275438689646b419735cf2e: Sending DeleteTablet for 3 replicas of tablet c30b5adbae424066a79035484016bc75
I20260926 20:53:35.837919 19961 tablet_service.cc:1514] Processing DeleteTablet for tablet c30b5adbae424066a79035484016bc75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:35 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:35992
I20260926 20:53:35.837883 19827 tablet_service.cc:1514] Processing DeleteTablet for tablet c30b5adbae424066a79035484016bc75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:35 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:35234
I20260926 20:53:35.838016 19693 tablet_service.cc:1514] Processing DeleteTablet for tablet c30b5adbae424066a79035484016bc75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:53:35 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:33390
I20260926 20:53:35.839440 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 19630
I20260926 20:53:35.850951 20068 tablet_replica.cc:330] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9: stopping tablet replica
I20260926 20:53:35.851334 20069 tablet_replica.cc:330] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3: stopping tablet replica
I20260926 20:53:35.851495 20068 raft_consensus.cc:2229] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:53:35.851828 20069 raft_consensus.cc:2229] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:35.852281 20068 raft_consensus.cc:2258] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:35.852413 20069 raft_consensus.cc:2258] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:35.853398 20070 tablet_replica.cc:330] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77: stopping tablet replica
I20260926 20:53:35.854067 20070 raft_consensus.cc:2229] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:53:35.854606 20070 raft_consensus.cc:2258] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:53:35.854875 20068 ts_tablet_manager.cc:1905] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:35.855088 20069 ts_tablet_manager.cc:1905] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:35.856753 20070 ts_tablet_manager.cc:1905] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:53:35.861934 20069 ts_tablet_manager.cc:1918] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:35.862311 20068 ts_tablet_manager.cc:1918] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:35.862581 20068 log.cc:1196] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/c30b5adbae424066a79035484016bc75
Sep 26 20:53:35 dist-test-slave-2dg3 krb5kdc[19524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:53:35.863302 20068 ts_tablet_manager.cc:1939] T c30b5adbae424066a79035484016bc75 P e17f6873c01448bba6daa3ad946a8fd9: Deleting consensus metadata
I20260926 20:53:35.863626 20070 ts_tablet_manager.cc:1918] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:53:35.863852 20070 log.cc:1196] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/c30b5adbae424066a79035484016bc75
I20260926 20:53:35.864434 20070 ts_tablet_manager.cc:1939] T c30b5adbae424066a79035484016bc75 P a51dc912c06742e1894e007e40f0ac77: Deleting consensus metadata
I20260926 20:53:35.864782 20069 log.cc:1196] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/c30b5adbae424066a79035484016bc75
I20260926 20:53:35.865363 20069 ts_tablet_manager.cc:1939] T c30b5adbae424066a79035484016bc75 P 3f64949b71564d8eb7b5ba8a475ae3c3: Deleting consensus metadata
I20260926 20:53:35.865334 19553 catalog_manager.cc:4841] TS e17f6873c01448bba6daa3ad946a8fd9 (127.8.249.131:45571): tablet c30b5adbae424066a79035484016bc75 (table test-table [id=deee3e7a2c254d8b9aefcc80f3e1e474]) successfully deleted
I20260926 20:53:35.866281 19553 catalog_manager.cc:4841] TS a51dc912c06742e1894e007e40f0ac77 (127.8.249.129:45041): tablet c30b5adbae424066a79035484016bc75 (table test-table [id=deee3e7a2c254d8b9aefcc80f3e1e474]) successfully deleted
I20260926 20:53:35.867290 19556 catalog_manager.cc:4841] TS 3f64949b71564d8eb7b5ba8a475ae3c3 (127.8.249.130:40123): tablet c30b5adbae424066a79035484016bc75 (table test-table [id=deee3e7a2c254d8b9aefcc80f3e1e474]) successfully deleted
I20260926 20:53:35.871853 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.841260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.8.249.129:45041)
0926 20:53:35.841487 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:35.841497 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:35.852658 (+ 11161us) server_negotiation.cc:407] Connection header received
0926 20:53:35.852998 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:35.853004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:35.853112 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:35.853254 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:35.854815 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.855972 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.857791 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.858474 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.858578 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.864615 (+ 6037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:35.864631 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:35.864648 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:35.864695 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:35.868678 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:35.869448 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:35.869458 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:35.869462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:35.869635 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:35.870072 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:35.870078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:35.870084 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:35.870545 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:35.870905 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:35.871211 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 20:53:35.871446 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:53:35.872592 19753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:35.969302 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 19630
I20260926 20:53:35.988023 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 19764
Sep 26 20:53:35 dist-test-slave-2dg3 krb5kdc[19524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:53:36.005465 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:35.988907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.8.249.130:40123)
0926 20:53:35.989112 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:35.989119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:35.989141 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:53:35.989208 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:35.989214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:35.989299 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:35.989420 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:35.990743 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.991933 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.993151 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:35.993779 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:35.993907 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:35.998472 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:35.998494 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:35.998515 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:35.998566 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:36.002433 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:36.003346 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:36.003356 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:36.003360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:36.003515 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:36.003864 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:36.003869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:36.003873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:36.004358 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:36.004672 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:36.004856 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 20:53:36.005085 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:53:36.006258 19887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:36.097610 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 19764
I20260926 20:53:36.116477 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 19898
Sep 26 20:53:36 dist-test-slave-2dg3 krb5kdc[19524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:53:36.135509 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:53:36.117416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.8.249.131:45571)
0926 20:53:36.117649 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 20:53:36.117654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:53:36.117676 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:53:36.118025 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:53:36.118032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:53:36.118144 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:53:36.118379 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:53:36.120200 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:36.121056 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:36.123012 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:53:36.123506 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:53:36.123617 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:53:36.129243 (+ 5626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:53:36.129255 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:53:36.129274 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:53:36.129314 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:53:36.132395 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:36.133127 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:36.133135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:36.133137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:36.133293 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:53:36.133689 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:53:36.133698 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:53:36.133700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:53:36.134063 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:53:36.134362 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 20:53:36.134681 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 20:53:36.135208 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260926 20:53:36.136274 20021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:36.256368 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 19898
I20260926 20:53:36.275703 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 19537
I20260926 20:53:36.276904 19599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:53:36.384989 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 19537
2026-09-26T20:53:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4452 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:53:36 dist-test-slave-2dg3 krb5kdc[20078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:53:36 dist-test-slave-2dg3 krb5kdc[20078](info): set up 2 sockets
Sep 26 20:53:36 dist-test-slave-2dg3 krb5kdc[20078](info): commencing operation
krb5kdc: starting...
W20260926 20:53:37.487789 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:53:37 dist-test-slave-2dg3 krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:53:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/rangeradmin_127.8.249.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/rangeradmin_127.8.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/rangerlookup_127.8.249.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/rangerlookup_127.8.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/HTTP_127.8.249.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/HTTP_127.8.249.148@KRBTEST.COM.keytab.
I20260926 20:53:37.809101 9190 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpGD2b0/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/postgres -l logfile start
2026-09-26 20:53:40.187 UTC [20108] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:53:40.187 UTC [20108] LOG: listening on IPv4 address "127.8.249.148", port 32925
2026-09-26 20:53:40.190 UTC [20108] LOG: listening on Unix socket "/tmp/.s.PGSQL.32925"
2026-09-26 20:53:40.213 UTC [20111] LOG: database system was shut down at 2026-09-26 20:53:38 UTC
2026-09-26 20:53:40.216 UTC [20108] LOG: database system is ready to accept connections
I20260926 20:53:41.187359 9190 mini_postgres.cc:96] Postgres bound to 32925
2026-09-26 20:53:41.205 UTC [20119] FATAL: database "slave" does not exist
127.8.249.148:32925 - accepting connections
I20260926 20:53:41.205797 9190 mini_ranger.cc:161] Starting Ranger...
I20260926 20:53:41.235105 9190 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:53:41.680541 9190 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:53:41.680651 9190 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-admin
2026-09-26 20:53:41,968 [I] DB FLAVOR :POSTGRES
2026-09-26 20:53:41,969 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:53:41,969 [I] Checking connection..
2026-09-26 20:53:41,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:53:42,224 [I] Checking connection passed.
2026-09-26 20:53:42,225 [I] --------- Verifying version history table ---------
2026-09-26 20:53:42,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:53:42,466 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:53:42,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:53:42,718 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:53:42,719 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:53:42,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:53:42.964 UTC [20183] WARNING: there is no transaction in progress
2026-09-26 20:53:42,970 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:53:42,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:53:43,240 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:53:43,240 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:53:43,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:53:43,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:53:43,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2dg3.c.gcp-upstream.internal','N') ;"
2026-09-26 20:53:43,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:53:44,241 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:53:44,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:53:44,495 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:53:44,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:53:44,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:53:44,972 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:53:44,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:53:45.902 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:45.917 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:45.933 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:45.949 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:45.966 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:45.983 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:46.037 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:46.043 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:46.058 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:46.068 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:46.079 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:46.093 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:46.103 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:46.109 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:46.115 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:46.122 UTC [20291] WARNING: there is no transaction in progress
2026-09-26 20:53:46,755 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:53:46,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:53:46,994 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:53:46,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:53:47,226 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:53:47,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:53:47,487 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:53:47,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:53:47,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:53:47,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2dg3.c.gcp-upstream.internal';"
2026-09-26 20:53:48,191 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:53:48,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:53:48,418 [I] DB_PATCHES have already been applied
I20260926 20:53:48.424546 9190 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpGD2b0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:53:48.424636 9190 mini_ranger.cc:193] Using host: 127.8.249.148
I20260926 20:53:48.439723 9190 mini_ranger.cc:227] Ranger admin URL: http://127.8.249.148:43751
Sep 26, 2026 8:53:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:53:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:53:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:53:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:53:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:53:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:53:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:53:49.593968 9190 mini_ranger.cc:160] Time spent starting Ranger: real 8.388s user 0.002s sys 0.040s
Sep 26, 2026 8:54:08 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:54:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 8:54:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:54:08 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:54:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:54:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:54:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:54:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:54:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:54:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:54:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:54:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:54:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:54:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:54:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:54:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:54:08 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:54:09 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:54:09.967098 9190 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/rangerkms_127.8.249.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/rangerkms_127.8.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.249.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.249.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.249.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/HTTP_127.8.249.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.249.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/HTTP_127.8.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:54:10 dist-test-slave-2dg3 krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456050, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:54:10.130144 9190 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:54:10.154847 9190 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:54:11.023056 9190 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:54:11.023226 9190 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-kms
I20260926 20:54:11.023345 9190 mini_ranger_kms.cc:227] Using postgres at 127.8.249.148:32925
2026-09-26 20:54:11,444 [I] DB FLAVOR :POSTGRES
2026-09-26 20:54:11,444 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:54:11,444 [I] Checking connection
2026-09-26 20:54:11,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:54:11,675 [I] connection success
2026-09-26 20:54:11,676 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:54:11,676 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:54:11,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:54:11,919 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:54:11,919 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:54:11,919 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:54:11,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:32925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:54:12,186 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:54:12.615979 9190 mini_ranger_kms.cc:310] Created kms service
I20260926 20:54:12.807132 9190 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:54:12.850010 9190 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:54:12 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456050, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:54:13.237861 9190 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:54:13.238207 9190 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:54:13.238265 9190 mini_ranger_kms.cc:240] Using host: 127.8.249.148
I20260926 20:54:13.259603 9190 mini_ranger_kms.cc:276] Ranger KMS PID: 20471
I20260926 20:54:13.259799 9190 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.249.148:50461
Sep 26, 2026 8:54:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:54:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:54:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:54:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:54:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:54:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 20:54:15 dist-test-slave-2dg3 krb5kdc[20078](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790456055, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.249.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:15.582633 9190 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.453s user 0.106s sys 0.346s
I20260926 20:54:15.582906 9190 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 20:54:15.583110 9190 mini_ranger_kms.cc:401] 127.8.249.148:50461/kms/v1/keys
Sep 26 20:54:16 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456055, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.249.148@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
Sep 26 20:54:16 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456055, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.249.148@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
Sep 26 20:54:18 dist-test-slave-2dg3 krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:54:18.732843 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:38501
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:38501
--ranger_config_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.249.148:50461/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:54:19.062263 20522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:19.062589 20522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:19.062738 20522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:19.069840 20522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:54:19.069983 20522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:19.070065 20522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:19.070152 20522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:54:19.070247 20522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:54:19.080154 20522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.249.148:50461/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:38501
--ranger_config_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:38501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:54:19.081403 20522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:19.082664 20522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:19.094384 20530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:19.094408 20527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:19.094391 20528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:19.095183 20522 server_base.cc:1034] running on GCE node
I20260926 20:54:19.096002 20522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:19.097448 20522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:19.098686 20522 hybrid_clock.cc:648] HybridClock initialized: now 1790456059098664 us; error 50 us; skew 500 ppm
I20260926 20:54:19.099095 20522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:54:19 dist-test-slave-2dg3 krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456059, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:19.106709 20522 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:54:19.108278 20522 webserver.cc:460] Webserver started at http://127.8.249.190:36707/ using document root <none> and password file <none>
I20260926 20:54:19.108976 20522 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:19.109125 20522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:19.109431 20522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:54:19 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456059, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:54:19.168057 20522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "1b46ded5a33241ae9315bd22a7621434"
format_stamp: "Formatted at 2026-09-26 20:54:19 on dist-test-slave-2dg3"
server_key: "82922407cc17046dbc3065eaf2adc09d"
server_key_iv: "661add103a0a63728adad92ebf5d0afb"
server_key_version: "kuduclusterkey@0"
I20260926 20:54:19.168956 20522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "1b46ded5a33241ae9315bd22a7621434"
format_stamp: "Formatted at 2026-09-26 20:54:19 on dist-test-slave-2dg3"
server_key: "82922407cc17046dbc3065eaf2adc09d"
server_key_iv: "661add103a0a63728adad92ebf5d0afb"
server_key_version: "kuduclusterkey@0"
I20260926 20:54:19.174491 20522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:54:19.191509 20537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:19.193286 20522 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:54:19.193512 20522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "1b46ded5a33241ae9315bd22a7621434"
format_stamp: "Formatted at 2026-09-26 20:54:19 on dist-test-slave-2dg3"
server_key: "82922407cc17046dbc3065eaf2adc09d"
server_key_iv: "661add103a0a63728adad92ebf5d0afb"
server_key_version: "kuduclusterkey@0"
I20260926 20:54:19.193755 20522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:19.209036 20522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:19.213845 20522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:19.214219 20522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:19.234269 20522 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:38501
I20260926 20:54:19.234292 20588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:38501 every 8 connection(s)
I20260926 20:54:19.235760 20522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:54:19.239481 20589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:19.243626 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 20522
I20260926 20:54:19.243834 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:54:19.244598 9190 external_mini_cluster.cc:1428] Setting key a8b80e2de63d2e47961a4fc0d887eab7
I20260926 20:54:19.250857 20589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434: Bootstrap starting.
I20260926 20:54:19.254169 20589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:19.255321 20589 log.cc:824] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:19.258574 20589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434: No bootstrap required, opened a new log
Sep 26 20:54:19 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:54:19.268810 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:19.246617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52702 (local address 127.8.249.190:38501)
0926 20:54:19.247460 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:19.247472 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:19.247510 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:54:19.249436 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:19.249461 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:19.249836 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:19.250146 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:19.251670 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:19.252961 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:19.254722 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:19.255412 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:19.255597 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:19.260138 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:19.260149 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:19.260172 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:19.260220 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:19.264047 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:19.264830 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:19.264839 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:19.264850 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:19.265038 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:19.265465 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:19.265471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:19.265477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:19.266227 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:19.266493 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:19.266946 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 20:54:19.267379 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":456,"threads_started":1}
I20260926 20:54:19.269275 20589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b46ded5a33241ae9315bd22a7621434" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 38501 } }
I20260926 20:54:19.269634 20589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:19.269742 20589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b46ded5a33241ae9315bd22a7621434, State: Initialized, Role: FOLLOWER
I20260926 20:54:19.270390 20589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b46ded5a33241ae9315bd22a7621434" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 38501 } }
I20260926 20:54:19.270686 20589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:54:19.270852 20589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:54:19.271037 20589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:19.273460 20589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b46ded5a33241ae9315bd22a7621434" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 38501 } }
I20260926 20:54:19.273969 20589 leader_election.cc:302] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b46ded5a33241ae9315bd22a7621434; no voters:
I20260926 20:54:19.274547 20589 leader_election.cc:288] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:54:19.274724 20594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:19.275440 20594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [term 1 LEADER]: Becoming Leader. State: Replica: 1b46ded5a33241ae9315bd22a7621434, State: Running, Role: LEADER
I20260926 20:54:19.276048 20594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b46ded5a33241ae9315bd22a7621434" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 38501 } }
I20260926 20:54:19.276901 20589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:54:19.279106 20595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b46ded5a33241ae9315bd22a7621434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b46ded5a33241ae9315bd22a7621434" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 38501 } } }
I20260926 20:54:19.279451 20595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:19.279891 20596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b46ded5a33241ae9315bd22a7621434. Latest consensus state: current_term: 1 leader_uuid: "1b46ded5a33241ae9315bd22a7621434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b46ded5a33241ae9315bd22a7621434" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 38501 } } }
I20260926 20:54:19.280189 20596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434 [sys.catalog]: This master's current role is: LEADER
I20260926 20:54:19.281672 20602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:54:19.286902 20602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:54:19.287787 20589 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:54:19.311702 20602 catalog_manager.cc:1347] Generated new cluster ID: b9a4b5313aa34d6b9d118dc46852db55
I20260926 20:54:19.311843 20602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:54:19.331813 20602 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:54:19.333787 20602 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:54:19.344429 20602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434: Generated new TSK 0
I20260926 20:54:19.345554 20602 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:54:20.159436 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.249.148:50461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:38501
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:54:20 dist-test-slave-2dg3 krb5kdc[20078](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790456060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 20:54:20.611451 20634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:20.611872 20634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:20.612025 20634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:20.623922 20634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:20.624079 20634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:20.624277 20634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:54:20.640465 20634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.249.148:50461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:38501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:54:20.642280 20634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:20.644117 20634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:20.660481 20645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:20.660799 20647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:20.661345 20644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:20.661767 20634 server_base.cc:1034] running on GCE node
I20260926 20:54:20.662565 20634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:20.663969 20634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:20.665257 20634 hybrid_clock.cc:648] HybridClock initialized: now 1790456060665199 us; error 45 us; skew 500 ppm
I20260926 20:54:20.665632 20634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:54:20 dist-test-slave-2dg3 krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:20.669708 20634 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:54:20.671401 20634 webserver.cc:460] Webserver started at http://127.8.249.129:41771/ using document root <none> and password file <none>
I20260926 20:54:20.672123 20634 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:20.672245 20634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:20.672582 20634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:54:20 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:54:20.693812 20634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "7795eecdd34948a7b3d713b2c6b328ab"
format_stamp: "Formatted at 2026-09-26 20:54:20 on dist-test-slave-2dg3"
server_key: "72781912f34615fd62b5ebb676062c3a"
server_key_iv: "e6061bb2a1d9caa383bac00160ded9f5"
server_key_version: "kuduclusterkey@0"
I20260926 20:54:20.695550 20634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "7795eecdd34948a7b3d713b2c6b328ab"
format_stamp: "Formatted at 2026-09-26 20:54:20 on dist-test-slave-2dg3"
server_key: "72781912f34615fd62b5ebb676062c3a"
server_key_iv: "e6061bb2a1d9caa383bac00160ded9f5"
server_key_version: "kuduclusterkey@0"
I20260926 20:54:20.714170 20634 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.008s sys 0.008s
I20260926 20:54:20.758104 20654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:20.760427 20634 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.012s sys 0.006s
I20260926 20:54:20.760676 20634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "7795eecdd34948a7b3d713b2c6b328ab"
format_stamp: "Formatted at 2026-09-26 20:54:20 on dist-test-slave-2dg3"
server_key: "72781912f34615fd62b5ebb676062c3a"
server_key_iv: "e6061bb2a1d9caa383bac00160ded9f5"
server_key_version: "kuduclusterkey@0"
I20260926 20:54:20.761000 20634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:20.778834 20634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:20.785246 20634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:20.785622 20634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:20.786975 20634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:20.789639 20634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:20.789737 20634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:20.789862 20634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:20.789928 20634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:20.865739 20634 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:46749
I20260926 20:54:20.865741 20766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:46749 every 8 connection(s)
I20260926 20:54:20.867851 20634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:54:20.876976 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 20634
I20260926 20:54:20.877244 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:54:20.878227 9190 external_mini_cluster.cc:1428] Setting key 58523338d96c3fd7489fc19c5c2c0610
Sep 26 20:54:20 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:54:20.937546 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:20.874900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:38501 (local address 127.8.249.129:58785)
0926 20:54:20.879059 (+ 4159us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:20.879142 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:20.882405 (+ 3263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:20.909398 (+ 26993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:20.909421 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:20.910421 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:20.914473 (+ 4052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:20.914519 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:20.916928 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:20.916938 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:20.919992 (+ 3054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:20.920027 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:20.921884 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:20.921896 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:20.922338 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:20.923401 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:20.923472 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:20.927104 (+ 3632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:20.932186 (+ 5082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:20.932198 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:20.932222 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:20.932948 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:20.933485 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:20.933490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:20.933494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:20.933791 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:20.934625 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:20.934634 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:20.935142 (+ 508us) client_negotiation.cc:769] Sending connection context
0926 20:54:20.935490 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 20:54:20.936075 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3737,"thread_start_us":4226,"threads_started":1}
I20260926 20:54:20.938784 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:20.873900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:58785 (local address 127.8.249.190:38501)
0926 20:54:20.888927 (+ 15027us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:20.888937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:20.888973 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:54:20.889070 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:20.889079 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:20.889201 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:20.914833 (+ 25632us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:20.915099 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:20.916653 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:20.920324 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:20.921265 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:20.921428 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:20.927559 (+ 6131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:20.927567 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:20.927572 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:20.927614 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:20.931926 (+ 4312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:20.933134 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:20.933141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:20.933145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:20.933317 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:20.933935 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:20.933940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:20.933944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:20.934378 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:20.934530 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:20.938133 (+ 3603us) server_negotiation.cc:299] Negotiation successful
0926 20:54:20.938401 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14821,"thread_start_us":15512,"threads_started":1}
I20260926 20:54:20.941176 20767 heartbeater.cc:347] Connected to a master server at 127.8.249.190:38501
I20260926 20:54:20.941552 20767 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
I20260926 20:54:20.943004 20767 heartbeater.cc:511] Master 127.8.249.190:38501 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
I20260926 20:54:20.945940 20554 ts_manager.cc:194] Registered new tserver with Master: 7795eecdd34948a7b3d713b2c6b328ab (127.8.249.129:46749)
I20260926 20:54:20.947990 20554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:58785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:54:20.983134 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.249.148:50461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:38501
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:54:21 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
Sep 26 20:54:21 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
W20260926 20:54:21.508147 20773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:21.508489 20773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:21.508677 20773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:21.519382 20773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:21.519553 20773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:21.519747 20773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:54:21.532472 20773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.249.148:50461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:38501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:54:21.534376 20773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:21.536150 20773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:21.568562 20780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:21.577261 20781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:21.585817 20783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:21.589884 20773 server_base.cc:1034] running on GCE node
I20260926 20:54:21.590991 20773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:21.599447 20773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:21.600805 20773 hybrid_clock.cc:648] HybridClock initialized: now 1790456061600712 us; error 104 us; skew 500 ppm
I20260926 20:54:21.601197 20773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:54:21 dist-test-slave-2dg3 krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:21.604628 20773 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:54:21.607153 20773 webserver.cc:460] Webserver started at http://127.8.249.130:43377/ using document root <none> and password file <none>
I20260926 20:54:21.607959 20773 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:21.608112 20773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:21.608449 20773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:54:21 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:54:21.625505 20773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "e9138c29b9b44e598133c973d3405160"
format_stamp: "Formatted at 2026-09-26 20:54:21 on dist-test-slave-2dg3"
server_key: "b568e9d5d5a9b7ab22a65c886740240a"
server_key_iv: "d106c47ae74b1fa9da8c1d1d3532536f"
server_key_version: "kuduclusterkey@0"
I20260926 20:54:21.626847 20773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "e9138c29b9b44e598133c973d3405160"
format_stamp: "Formatted at 2026-09-26 20:54:21 on dist-test-slave-2dg3"
server_key: "b568e9d5d5a9b7ab22a65c886740240a"
server_key_iv: "d106c47ae74b1fa9da8c1d1d3532536f"
server_key_version: "kuduclusterkey@0"
I20260926 20:54:21.639595 20773 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.011s sys 0.000s
I20260926 20:54:21.660449 20797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:21.662489 20773 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:54:21.662774 20773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "e9138c29b9b44e598133c973d3405160"
format_stamp: "Formatted at 2026-09-26 20:54:21 on dist-test-slave-2dg3"
server_key: "b568e9d5d5a9b7ab22a65c886740240a"
server_key_iv: "d106c47ae74b1fa9da8c1d1d3532536f"
server_key_version: "kuduclusterkey@0"
I20260926 20:54:21.663199 20773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:21.678362 20773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:21.683022 20773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:21.683346 20773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:21.684520 20773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:21.686066 20773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:21.686196 20773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:21.686350 20773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:21.686426 20773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:21.732409 20773 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:46565
I20260926 20:54:21.732411 20909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:46565 every 8 connection(s)
I20260926 20:54:21.733758 20773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:54:21.739552 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 20773
I20260926 20:54:21.739755 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:54:21.740223 9190 external_mini_cluster.cc:1428] Setting key 9f42c3ffff839d81088c76a24d6a0e20
Sep 26 20:54:21 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:54:21.758280 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:21.736693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:53995 (local address 127.8.249.190:38501)
0926 20:54:21.737190 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:21.737198 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:21.738329 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 20:54:21.740538 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:21.740542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:21.740611 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:21.740734 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:21.742473 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:21.743558 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:21.746271 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:21.747348 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:21.747487 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:21.752028 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:21.752036 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:21.752041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:21.752089 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:21.755093 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:21.755849 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:21.755858 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:21.755861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:21.756003 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:21.756458 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:21.756464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:21.756467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:21.756762 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:21.756987 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:21.757668 (+ 681us) server_negotiation.cc:299] Negotiation successful
0926 20:54:21.757899 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":247,"threads_started":1}
I20260926 20:54:21.759456 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:21.737308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:38501 (local address 127.8.249.130:53995)
0926 20:54:21.738140 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:21.738186 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:21.740131 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:21.741035 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:21.741048 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:21.741551 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:21.742256 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:21.742301 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:21.743738 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:21.743746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:21.746074 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:21.746094 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:21.747609 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:21.747620 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:21.748002 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:21.748886 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:21.748931 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:21.751737 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:21.755247 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:21.755257 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:21.755275 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:21.755721 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:21.756126 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:21.756131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:21.756135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:21.756329 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:21.756963 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:21.756972 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:21.757310 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 20:54:21.757600 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 20:54:21.758131 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":365,"threads_started":1}
I20260926 20:54:21.761317 20910 heartbeater.cc:347] Connected to a master server at 127.8.249.190:38501
I20260926 20:54:21.761656 20910 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:21.762380 20910 heartbeater.cc:511] Master 127.8.249.190:38501 requested a full tablet report, sending...
I20260926 20:54:21.763967 20554 ts_manager.cc:194] Registered new tserver with Master: e9138c29b9b44e598133c973d3405160 (127.8.249.130:46565)
I20260926 20:54:21.764820 20554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:53995
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:54:21.804631 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.249.148:50461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:38501
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:54:21.952078 20767 heartbeater.cc:503] Master 127.8.249.190:38501 was elected leader, sending a full tablet report...
W20260926 20:54:22.189237 20916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:54:22.189549 20916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:54:22.189714 20916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:54:22.197777 20916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:54:22.197938 20916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:54:22.198135 20916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:54:22.208811 20916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.249.148:50461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:38501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:54:22.210073 20916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:54:22.211390 20916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:54:22.223989 20921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:22.224089 20924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:54:22.224128 20922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:54:22.225311 20916 server_base.cc:1034] running on GCE node
I20260926 20:54:22.225862 20916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:54:22.226744 20916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:54:22.227986 20916 hybrid_clock.cc:648] HybridClock initialized: now 1790456062227947 us; error 37 us; skew 500 ppm
I20260926 20:54:22.228251 20916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:54:22 dist-test-slave-2dg3 krb5kdc[20078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456062, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:54:22.231699 20916 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:54:22.233206 20916 webserver.cc:460] Webserver started at http://127.8.249.131:43637/ using document root <none> and password file <none>
I20260926 20:54:22.234155 20916 fs_manager.cc:359] Metadata directory not provided
I20260926 20:54:22.234266 20916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:54:22.234512 20916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:54:22 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456062, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:54:22.250496 20916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "7760c265afc54938ad027c88eb11ca95"
format_stamp: "Formatted at 2026-09-26 20:54:22 on dist-test-slave-2dg3"
server_key: "729e85d19b7ed4ce9748994938df6c7d"
server_key_iv: "89f80405f4b01481143cc8b18548ccfd"
server_key_version: "kuduclusterkey@0"
I20260926 20:54:22.251425 20916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "7760c265afc54938ad027c88eb11ca95"
format_stamp: "Formatted at 2026-09-26 20:54:22 on dist-test-slave-2dg3"
server_key: "729e85d19b7ed4ce9748994938df6c7d"
server_key_iv: "89f80405f4b01481143cc8b18548ccfd"
server_key_version: "kuduclusterkey@0"
I20260926 20:54:22.256821 20916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:54:22.274299 20931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:22.276139 20916 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:54:22.276329 20916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "7760c265afc54938ad027c88eb11ca95"
format_stamp: "Formatted at 2026-09-26 20:54:22 on dist-test-slave-2dg3"
server_key: "729e85d19b7ed4ce9748994938df6c7d"
server_key_iv: "89f80405f4b01481143cc8b18548ccfd"
server_key_version: "kuduclusterkey@0"
I20260926 20:54:22.276618 20916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:54:22.300235 20916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:54:22.305648 20916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:54:22.306041 20916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:54:22.307296 20916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:54:22.309273 20916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:54:22.309368 20916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:22.309494 20916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:54:22.309563 20916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:54:22.347572 20916 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:39391
I20260926 20:54:22.347539 21043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:39391 every 8 connection(s)
I20260926 20:54:22.349246 20916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:54:22.353173 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 20916
I20260926 20:54:22.353344 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:54:22.353873 9190 external_mini_cluster.cc:1428] Setting key 58b4affbb154fee4bd62b36312f54657
Sep 26 20:54:22 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456062, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:54:22.373591 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.351970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:33019 (local address 127.8.249.190:38501)
0926 20:54:22.352467 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:22.352473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:22.353863 (+ 1390us) server_negotiation.cc:407] Connection header received
0926 20:54:22.356003 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:22.356010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:22.356109 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:22.356221 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:22.358193 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.359033 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.361433 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.361844 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.361932 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.366078 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:22.366083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:22.366086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:22.366128 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:22.370355 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:22.371167 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:22.371173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:22.371176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:22.371331 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:22.371780 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:22.371785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:22.371789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:22.372166 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:22.372318 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:22.373004 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 20:54:22.373227 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":244,"threads_started":1}
I20260926 20:54:22.374365 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.352456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:38501 (local address 127.8.249.131:33019)
0926 20:54:22.353265 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:22.353317 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:22.355677 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:22.356509 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:22.356521 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:22.357138 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:54:22.358007 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:22.358035 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.359200 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.359206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:22.361290 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:22.361305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.362084 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.362091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:22.362400 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.363180 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:54:22.363210 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:54:22.365834 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:54:22.370498 (+ 4664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:22.370505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:22.370520 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:22.371042 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:22.371440 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:54:22.371444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:54:22.371446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:54:22.371661 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:54:22.372288 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:54:22.372294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:54:22.372613 (+ 319us) client_negotiation.cc:769] Sending connection context
0926 20:54:22.372837 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 20:54:22.373233 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":326,"threads_started":1}
I20260926 20:54:22.375888 21044 heartbeater.cc:347] Connected to a master server at 127.8.249.190:38501
I20260926 20:54:22.376214 21044 heartbeater.cc:464] Registering TS with master...
I20260926 20:54:22.377010 21044 heartbeater.cc:511] Master 127.8.249.190:38501 requested a full tablet report, sending...
I20260926 20:54:22.378654 20554 ts_manager.cc:194] Registered new tserver with Master: 7760c265afc54938ad027c88eb11ca95 (127.8.249.131:39391)
I20260926 20:54:22.379485 20554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:33019
I20260926 20:54:22.382191 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:54:22.403069 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.388134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.8.249.190:38501)
0926 20:54:22.388325 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:22.388332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:22.388752 (+ 420us) server_negotiation.cc:407] Connection header received
0926 20:54:22.388914 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:22.388920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:22.389025 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:22.389147 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:22.390616 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.391822 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.393615 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.394245 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.394384 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.396903 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:22.396910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:22.396915 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:22.396958 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:22.400359 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:22.401061 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:22.401066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:22.401070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:22.401240 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:22.401606 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:22.401611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:22.401615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:22.401982 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:22.402219 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:22.402487 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 20:54:22.402705 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:54:22.407351 20554 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:54:22.414489 20554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:54:22.455044 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.428356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:46565 (local address 127.0.0.1:44376)
0926 20:54:22.442510 (+ 14154us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:22.442574 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:22.442770 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:22.445035 (+ 2265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:22.445046 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:22.445085 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:22.446129 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:22.446152 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.448816 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.448825 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:22.451814 (+ 2989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:22.451827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.454021 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.454028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:22.454221 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.454274 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 20:54:22.454426 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 20:54:22.454647 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13802,"thread_start_us":15198,"threads_started":1}
I20260926 20:54:22.455964 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.428910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.8.249.130:46565)
0926 20:54:22.444218 (+ 15308us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:22.444232 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:22.444273 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:54:22.444399 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:22.444411 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:22.444757 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:22.445016 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:22.446514 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.448605 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.451997 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.453493 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.453654 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.453870 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:22.455450 (+ 1580us) server_negotiation.cc:299] Negotiation successful
0926 20:54:22.455666 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15082,"thread_start_us":16268,"threads_started":1}
I20260926 20:54:22.456557 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.439093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:46749 (local address 127.0.0.1:43520)
0926 20:54:22.447181 (+ 8088us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:22.447202 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:22.447307 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:22.448229 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:22.448234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:22.448294 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:22.449444 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:22.449459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.451455 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.451464 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:22.453504 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:22.453517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.455724 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.455730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:22.455916 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.455957 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 20:54:22.456051 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:54:22.456199 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7089,"thread_start_us":6433,"threads_started":1}
I20260926 20:54:22.457372 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.439419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.8.249.129:46749)
0926 20:54:22.445956 (+ 6537us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:22.445966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:22.447423 (+ 1457us) server_negotiation.cc:407] Connection header received
0926 20:54:22.447523 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:22.447531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:22.447782 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:22.447981 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:22.449696 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.451293 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.453672 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.455192 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.455309 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.455530 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:22.456864 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0926 20:54:22.457047 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6358,"thread_start_us":7381,"threads_started":1}
I20260926 20:54:22.459985 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.428518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:39391 (local address 127.0.0.1:36100)
0926 20:54:22.443380 (+ 14862us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:22.443396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:22.443489 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:22.450070 (+ 6581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:22.450075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:22.450097 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:22.450489 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:22.450500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.453779 (+ 3279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.453784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:22.455865 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:22.455891 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.458737 (+ 2846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.458747 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:22.458943 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.458978 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:54:22.459047 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:54:22.459179 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14755,"mutex_wait_us":2,"thread_start_us":14651,"threads_started":1}
I20260926 20:54:22.461402 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.429077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.8.249.131:39391)
0926 20:54:22.449513 (+ 20436us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:22.449522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:22.449551 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:54:22.449634 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:22.449641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:22.449857 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:22.450685 (+ 828us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:22.452110 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.453615 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.456078 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.458460 (+ 2382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.460610 (+ 2150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.460844 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:22.460929 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:54:22.461085 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20273,"thread_start_us":20146,"threads_started":1}
I20260926 20:54:22.460887 20845 tablet_service.cc:1467] Processing CreateTablet for tablet 0aaa4c6754ca4d27b86987b60fb9fc5f (DEFAULT_TABLE table=test-table [id=91b2993c98124596855430b64e6b6504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:22.461256 20702 tablet_service.cc:1467] Processing CreateTablet for tablet 0aaa4c6754ca4d27b86987b60fb9fc5f (DEFAULT_TABLE table=test-table [id=91b2993c98124596855430b64e6b6504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:22.462375 20845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aaa4c6754ca4d27b86987b60fb9fc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:22.462706 20702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aaa4c6754ca4d27b86987b60fb9fc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:22.464953 20979 tablet_service.cc:1467] Processing CreateTablet for tablet 0aaa4c6754ca4d27b86987b60fb9fc5f (DEFAULT_TABLE table=test-table [id=91b2993c98124596855430b64e6b6504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:54:22.466344 20979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aaa4c6754ca4d27b86987b60fb9fc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:54:22.489467 21064 tablet_bootstrap.cc:492] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab: Bootstrap starting.
I20260926 20:54:22.491197 21065 tablet_bootstrap.cc:492] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95: Bootstrap starting.
I20260926 20:54:22.495198 21064 tablet_bootstrap.cc:654] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:22.495198 21065 tablet_bootstrap.cc:654] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:22.495584 21063 tablet_bootstrap.cc:492] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160: Bootstrap starting.
I20260926 20:54:22.496680 21064 log.cc:824] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:22.497254 21065 log.cc:824] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:22.499337 21063 tablet_bootstrap.cc:654] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160: Neither blocks nor log segments found. Creating new log.
I20260926 20:54:22.501077 21063 log.cc:824] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160: Log is configured to *not* fsync() on all Append() calls
I20260926 20:54:22.507177 21065 tablet_bootstrap.cc:492] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95: No bootstrap required, opened a new log
I20260926 20:54:22.507495 21065 ts_tablet_manager.cc:1397] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.007s
I20260926 20:54:22.508165 21064 tablet_bootstrap.cc:492] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab: No bootstrap required, opened a new log
I20260926 20:54:22.508430 21064 ts_tablet_manager.cc:1397] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.004s
I20260926 20:54:22.518599 21065 raft_consensus.cc:348] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9138c29b9b44e598133c973d3405160" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46565 } } peers { permanent_uuid: "7760c265afc54938ad027c88eb11ca95" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39391 } } peers { permanent_uuid: "7795eecdd34948a7b3d713b2c6b328ab" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 46749 } }
I20260926 20:54:22.520177 21065 raft_consensus.cc:374] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:22.520311 21065 raft_consensus.cc:729] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7760c265afc54938ad027c88eb11ca95, State: Initialized, Role: FOLLOWER
I20260926 20:54:22.521023 21065 consensus_queue.cc:260] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9138c29b9b44e598133c973d3405160" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46565 } } peers { permanent_uuid: "7760c265afc54938ad027c88eb11ca95" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39391 } } peers { permanent_uuid: "7795eecdd34948a7b3d713b2c6b328ab" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 46749 } }
I20260926 20:54:22.521242 21064 raft_consensus.cc:348] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9138c29b9b44e598133c973d3405160" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46565 } } peers { permanent_uuid: "7760c265afc54938ad027c88eb11ca95" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39391 } } peers { permanent_uuid: "7795eecdd34948a7b3d713b2c6b328ab" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 46749 } }
I20260926 20:54:22.521795 21064 raft_consensus.cc:374] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:22.521952 21064 raft_consensus.cc:729] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7795eecdd34948a7b3d713b2c6b328ab, State: Initialized, Role: FOLLOWER
I20260926 20:54:22.522107 21063 tablet_bootstrap.cc:492] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160: No bootstrap required, opened a new log
I20260926 20:54:22.522467 21063 ts_tablet_manager.cc:1397] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.017s
I20260926 20:54:22.522910 21064 consensus_queue.cc:260] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9138c29b9b44e598133c973d3405160" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46565 } } peers { permanent_uuid: "7760c265afc54938ad027c88eb11ca95" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39391 } } peers { permanent_uuid: "7795eecdd34948a7b3d713b2c6b328ab" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 46749 } }
I20260926 20:54:22.531919 21064 ts_tablet_manager.cc:1428] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab: Time spent starting tablet: real 0.023s user 0.016s sys 0.004s
I20260926 20:54:22.532034 21065 ts_tablet_manager.cc:1428] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95: Time spent starting tablet: real 0.024s user 0.015s sys 0.008s
I20260926 20:54:22.533185 21063 raft_consensus.cc:348] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9138c29b9b44e598133c973d3405160" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46565 } } peers { permanent_uuid: "7760c265afc54938ad027c88eb11ca95" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39391 } } peers { permanent_uuid: "7795eecdd34948a7b3d713b2c6b328ab" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 46749 } }
I20260926 20:54:22.533685 21063 raft_consensus.cc:374] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:54:22.533798 21063 raft_consensus.cc:729] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9138c29b9b44e598133c973d3405160, State: Initialized, Role: FOLLOWER
I20260926 20:54:22.534425 21044 heartbeater.cc:503] Master 127.8.249.190:38501 was elected leader, sending a full tablet report...
I20260926 20:54:22.536098 21063 consensus_queue.cc:260] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9138c29b9b44e598133c973d3405160" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46565 } } peers { permanent_uuid: "7760c265afc54938ad027c88eb11ca95" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39391 } } peers { permanent_uuid: "7795eecdd34948a7b3d713b2c6b328ab" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 46749 } }
I20260926 20:54:22.538962 20910 heartbeater.cc:503] Master 127.8.249.190:38501 was elected leader, sending a full tablet report...
I20260926 20:54:22.539677 21063 ts_tablet_manager.cc:1428] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260926 20:54:22.566849 21071 raft_consensus.cc:482] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:54:22.567256 21071 raft_consensus.cc:504] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9138c29b9b44e598133c973d3405160" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46565 } } peers { permanent_uuid: "7760c265afc54938ad027c88eb11ca95" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39391 } } peers { permanent_uuid: "7795eecdd34948a7b3d713b2c6b328ab" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 46749 } }
I20260926 20:54:22.569244 21071 leader_election.cc:288] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7760c265afc54938ad027c88eb11ca95 (127.8.249.131:39391), 7795eecdd34948a7b3d713b2c6b328ab (127.8.249.129:46749)
I20260926 20:54:22.577368 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.569797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:41343 (local address 127.8.249.131:39391)
0926 20:54:22.570068 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:22.570077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:22.570834 (+ 757us) server_negotiation.cc:407] Connection header received
0926 20:54:22.571092 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:22.571097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:22.571174 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:22.571260 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:22.572062 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.572934 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.575318 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.576418 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.576571 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.576638 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:22.577025 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 20:54:22.577146 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 20:54:22.577400 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.569405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:39391 (local address 127.8.249.130:41343)
0926 20:54:22.570664 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:22.570694 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:22.570910 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:22.571414 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:22.571419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:22.571449 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:22.571845 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:22.571879 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.573076 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.573079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:22.575155 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:22.575170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.576672 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.576678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:22.576879 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.576903 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:54:22.576976 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:54:22.577074 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":1,"thread_start_us":1785,"threads_started":1}
I20260926 20:54:22.578547 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.570165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:43027 (local address 127.8.249.129:46749)
0926 20:54:22.570346 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:22.570352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:22.571566 (+ 1214us) server_negotiation.cc:407] Connection header received
0926 20:54:22.571724 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:22.571729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:22.571800 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:22.571888 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:54:22.572445 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.573489 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.575671 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.576432 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.576634 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.576710 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:22.578175 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0926 20:54:22.578319 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:54:22.578565 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.569689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:46749 (local address 127.8.249.130:43027)
0926 20:54:22.571454 (+ 1765us) negotiation.cc:107] Waiting for socket to connect
0926 20:54:22.571470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:54:22.571608 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:54:22.571920 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:54:22.571927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:54:22.571955 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:54:22.572288 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:22.572301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.573635 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.573640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:22.575527 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:54:22.575536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.577762 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.577768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:54:22.577968 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.578018 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 20:54:22.578129 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:54:22.578258 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1636,"thread_start_us":647,"threads_started":1}
I20260926 20:54:22.578568 20999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aaa4c6754ca4d27b86987b60fb9fc5f" candidate_uuid: "e9138c29b9b44e598133c973d3405160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7760c265afc54938ad027c88eb11ca95" is_pre_election: true
I20260926 20:54:22.579144 20999 raft_consensus.cc:2454] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9138c29b9b44e598133c973d3405160 in term 0.
I20260926 20:54:22.580286 20799 leader_election.cc:302] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7760c265afc54938ad027c88eb11ca95, e9138c29b9b44e598133c973d3405160; no voters:
I20260926 20:54:22.579722 20722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aaa4c6754ca4d27b86987b60fb9fc5f" candidate_uuid: "e9138c29b9b44e598133c973d3405160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7795eecdd34948a7b3d713b2c6b328ab" is_pre_election: true
I20260926 20:54:22.580716 21071 raft_consensus.cc:2781] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:54:22.580706 20722 raft_consensus.cc:2454] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9138c29b9b44e598133c973d3405160 in term 0.
I20260926 20:54:22.580864 21071 raft_consensus.cc:482] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:54:22.580961 21071 raft_consensus.cc:3037] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:22.583994 21071 raft_consensus.cc:504] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9138c29b9b44e598133c973d3405160" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46565 } } peers { permanent_uuid: "7760c265afc54938ad027c88eb11ca95" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39391 } } peers { permanent_uuid: "7795eecdd34948a7b3d713b2c6b328ab" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 46749 } }
I20260926 20:54:22.584637 21071 leader_election.cc:288] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [CANDIDATE]: Term 1 election: Requested vote from peers 7760c265afc54938ad027c88eb11ca95 (127.8.249.131:39391), 7795eecdd34948a7b3d713b2c6b328ab (127.8.249.129:46749)
I20260926 20:54:22.585103 20999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aaa4c6754ca4d27b86987b60fb9fc5f" candidate_uuid: "e9138c29b9b44e598133c973d3405160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7760c265afc54938ad027c88eb11ca95"
I20260926 20:54:22.585098 20722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aaa4c6754ca4d27b86987b60fb9fc5f" candidate_uuid: "e9138c29b9b44e598133c973d3405160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7795eecdd34948a7b3d713b2c6b328ab"
I20260926 20:54:22.585294 20722 raft_consensus.cc:3037] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:22.585316 20999 raft_consensus.cc:3037] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:54:22.587752 20999 raft_consensus.cc:2454] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9138c29b9b44e598133c973d3405160 in term 1.
I20260926 20:54:22.588172 20799 leader_election.cc:302] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7760c265afc54938ad027c88eb11ca95, e9138c29b9b44e598133c973d3405160; no voters:
I20260926 20:54:22.588299 20722 raft_consensus.cc:2454] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9138c29b9b44e598133c973d3405160 in term 1.
I20260926 20:54:22.588492 21071 raft_consensus.cc:2781] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:54:22.589043 21071 raft_consensus.cc:686] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [term 1 LEADER]: Becoming Leader. State: Replica: e9138c29b9b44e598133c973d3405160, State: Running, Role: LEADER
I20260926 20:54:22.589620 21071 consensus_queue.cc:237] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9138c29b9b44e598133c973d3405160" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46565 } } peers { permanent_uuid: "7760c265afc54938ad027c88eb11ca95" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39391 } } peers { permanent_uuid: "7795eecdd34948a7b3d713b2c6b328ab" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 46749 } }
I20260926 20:54:22.594483 20553 catalog_manager.cc:5495] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9138c29b9b44e598133c973d3405160 (127.8.249.130). New cstate: current_term: 1 leader_uuid: "e9138c29b9b44e598133c973d3405160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9138c29b9b44e598133c973d3405160" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7760c265afc54938ad027c88eb11ca95" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7795eecdd34948a7b3d713b2c6b328ab" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 46749 } health_report { overall_health: UNKNOWN } } }
W20260926 20:54:22.600867 21045 tablet.cc:2369] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:54:22.640201 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.634652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.8.249.130:46565)
0926 20:54:22.634859 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:22.634867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:22.634887 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:54:22.635185 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:22.635189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:22.635269 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:22.635433 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:54:22.635808 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.636503 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.637955 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.638356 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.638499 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.639044 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:54:22.639562 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:54:22.639724 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:22.639811 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 20:54:22.639899 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:54:22.648427 20999 raft_consensus.cc:1261] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95 [term 1 FOLLOWER]: Refusing update from remote peer e9138c29b9b44e598133c973d3405160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:22.648566 20722 raft_consensus.cc:1261] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab [term 1 FOLLOWER]: Refusing update from remote peer e9138c29b9b44e598133c973d3405160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:54:22.649577 21074 consensus_queue.cc:1035] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7795eecdd34948a7b3d713b2c6b328ab" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 46749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:22.650004 21071 consensus_queue.cc:1035] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7760c265afc54938ad027c88eb11ca95" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:54:22.693408 20554 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:22.693776 20554 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:54:22.697312 20554 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b46ded5a33241ae9315bd22a7621434: Sending DeleteTablet for 3 replicas of tablet 0aaa4c6754ca4d27b86987b60fb9fc5f
I20260926 20:54:22.698457 20845 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aaa4c6754ca4d27b86987b60fb9fc5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:22 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:44376
I20260926 20:54:22.699345 20702 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aaa4c6754ca4d27b86987b60fb9fc5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:22 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:43520
I20260926 20:54:22.699746 20979 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aaa4c6754ca4d27b86987b60fb9fc5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:54:22 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:36100
I20260926 20:54:22.703258 21088 tablet_replica.cc:330] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160: stopping tablet replica
I20260926 20:54:22.703691 21088 raft_consensus.cc:2229] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:54:22.704315 21088 raft_consensus.cc:2258] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:22.704312 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 20634
I20260926 20:54:22.705052 21090 tablet_replica.cc:330] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95: stopping tablet replica
I20260926 20:54:22.705436 21090 raft_consensus.cc:2229] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:54:22.705940 21090 raft_consensus.cc:2258] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:22.709821 21089 tablet_replica.cc:330] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab: stopping tablet replica
I20260926 20:54:22.710250 21089 raft_consensus.cc:2229] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:54:22.710779 21089 raft_consensus.cc:2258] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:54:22.713044 21089 ts_tablet_manager.cc:1905] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:54:22.715513 21090 ts_tablet_manager.cc:1905] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:54:22.715788 21088 ts_tablet_manager.cc:1905] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:54:22.720244 21089 ts_tablet_manager.cc:1918] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:54:22.720432 21089 log.cc:1196] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/0aaa4c6754ca4d27b86987b60fb9fc5f
I20260926 20:54:22.720961 21089 ts_tablet_manager.cc:1939] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7795eecdd34948a7b3d713b2c6b328ab: Deleting consensus metadata
I20260926 20:54:22.723052 20541 catalog_manager.cc:4841] TS 7795eecdd34948a7b3d713b2c6b328ab (127.8.249.129:46749): tablet 0aaa4c6754ca4d27b86987b60fb9fc5f (table test-table [id=91b2993c98124596855430b64e6b6504]) successfully deleted
I20260926 20:54:22.728014 21088 ts_tablet_manager.cc:1918] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:54:22.728262 21088 log.cc:1196] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/0aaa4c6754ca4d27b86987b60fb9fc5f
I20260926 20:54:22.728834 21088 ts_tablet_manager.cc:1939] T 0aaa4c6754ca4d27b86987b60fb9fc5f P e9138c29b9b44e598133c973d3405160: Deleting consensus metadata
I20260926 20:54:22.729501 21090 ts_tablet_manager.cc:1918] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:54:22.729708 21090 log.cc:1196] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/0aaa4c6754ca4d27b86987b60fb9fc5f
I20260926 20:54:22.730247 21090 ts_tablet_manager.cc:1939] T 0aaa4c6754ca4d27b86987b60fb9fc5f P 7760c265afc54938ad027c88eb11ca95: Deleting consensus metadata
I20260926 20:54:22.731351 20540 catalog_manager.cc:4841] TS e9138c29b9b44e598133c973d3405160 (127.8.249.130:46565): tablet 0aaa4c6754ca4d27b86987b60fb9fc5f (table test-table [id=91b2993c98124596855430b64e6b6504]) successfully deleted
I20260926 20:54:22.732781 20539 catalog_manager.cc:4841] TS 7760c265afc54938ad027c88eb11ca95 (127.8.249.131:39391): tablet 0aaa4c6754ca4d27b86987b60fb9fc5f (table test-table [id=91b2993c98124596855430b64e6b6504]) successfully deleted
Sep 26 20:54:22 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:54:22.743824 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.709172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43526 (local address 127.8.249.129:46749)
0926 20:54:22.709509 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:22.709519 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:22.723412 (+ 13893us) server_negotiation.cc:407] Connection header received
0926 20:54:22.723544 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:22.723550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:22.723629 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:22.723748 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:22.725462 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.726587 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.728750 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.729385 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.729529 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.735698 (+ 6169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:22.735725 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:22.735746 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:22.735803 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:22.740008 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:22.741129 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:22.741142 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:22.741145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:22.741344 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:22.741852 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:22.741860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:22.741865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:22.742417 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:22.742751 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:22.743134 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 20:54:22.743389 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 20:54:22.745054 20762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:54:22.870875 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 20634
I20260926 20:54:22.888960 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 20773
Sep 26 20:54:22 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:54:22.905635 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:22.889801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.8.249.130:46565)
0926 20:54:22.890005 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:22.890013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:22.890032 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:54:22.890236 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:22.890242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:22.890332 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:22.890455 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:22.891801 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.892656 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.893994 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:22.894440 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:22.894579 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:22.899204 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:22.899219 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:22.899236 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:22.899286 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:22.902361 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:22.903437 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:22.903448 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:22.903452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:22.903640 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:22.904039 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:22.904052 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:22.904055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:22.904512 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:22.904861 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:22.905016 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 20:54:22.905247 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:54:22.906347 20905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:54:23.021613 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 20773
I20260926 20:54:23.039512 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 20916
Sep 26 20:54:23 dist-test-slave-2dg3 krb5kdc[20078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:54:23.054471 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:54:23.040286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.8.249.131:39391)
0926 20:54:23.040469 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 20:54:23.040473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:54:23.040491 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:54:23.040609 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:54:23.040613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:54:23.040680 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:54:23.040763 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:54:23.042128 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:23.042928 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:23.044175 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:54:23.044538 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:54:23.044628 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:54:23.049101 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:54:23.049115 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:54:23.049132 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:54:23.049167 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:54:23.051984 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:23.052689 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:23.052696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:23.052698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:23.052824 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:54:23.053145 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:54:23.053147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:54:23.053149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:54:23.053476 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:54:23.053686 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 20:54:23.053940 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 20:54:23.054098 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:54:23.055194 21039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:54:23.143821 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 20916
I20260926 20:54:23.159973 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 20522
I20260926 20:54:23.161159 20584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:54:23.274636 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 20522
I20260926 20:54:23.296475 9190 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:54:23.632820 9190 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:54:23.633009 9190 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:54:23.980190 9190 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:54:23.980 UTC [20108] LOG: received smart shutdown request
2026-09-26 20:54:23.984 UTC [20108] LOG: background worker "logical replication launcher" (PID 20117) exited with exit code 1
2026-09-26 20:54:23.984 UTC [20112] LOG: shutting down
2026-09-26 20:54:24.155 UTC [20108] LOG: database system is shut down
2026-09-26T20:54:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47900 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:54:24 dist-test-slave-2dg3 krb5kdc[21104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:54:24 dist-test-slave-2dg3 krb5kdc[21104](info): set up 2 sockets
Sep 26 20:54:24 dist-test-slave-2dg3 krb5kdc[21104](info): commencing operation
krb5kdc: starting...
W20260926 20:54:25.387166 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:54:25 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:54:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:54:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/rangeradmin_127.8.249.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/rangeradmin_127.8.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/rangerlookup_127.8.249.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/rangerlookup_127.8.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/HTTP_127.8.249.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/HTTP_127.8.249.148@KRBTEST.COM.keytab.
I20260926 20:54:25.713840 9190 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpGD2b0/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/postgres -l logfile start
2026-09-26 20:54:28.140 UTC [21134] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:54:28.140 UTC [21134] LOG: listening on IPv4 address "127.8.249.148", port 35511
2026-09-26 20:54:28.143 UTC [21134] LOG: listening on Unix socket "/tmp/.s.PGSQL.35511"
2026-09-26 20:54:28.164 UTC [21137] LOG: database system was shut down at 2026-09-26 20:54:26 UTC
2026-09-26 20:54:28.167 UTC [21134] LOG: database system is ready to accept connections
I20260926 20:54:29.130894 9190 mini_postgres.cc:96] Postgres bound to 35511
2026-09-26 20:54:29.147 UTC [21145] FATAL: database "slave" does not exist
127.8.249.148:35511 - accepting connections
I20260926 20:54:29.148514 9190 mini_ranger.cc:161] Starting Ranger...
I20260926 20:54:29.174887 9190 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:54:29.606056 9190 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:54:29.606176 9190 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-admin
2026-09-26 20:54:29,877 [I] DB FLAVOR :POSTGRES
2026-09-26 20:54:29,878 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:54:29,878 [I] Checking connection..
2026-09-26 20:54:29,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:54:30,147 [I] Checking connection passed.
2026-09-26 20:54:30,148 [I] --------- Verifying version history table ---------
2026-09-26 20:54:30,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:54:30,378 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:54:30,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:54:30,635 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:54:30,636 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:54:30,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:54:30.862 UTC [21209] WARNING: there is no transaction in progress
2026-09-26 20:54:30,868 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:54:30,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:54:31,115 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:54:31,115 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:54:31,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:54:31,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:54:31,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2dg3.c.gcp-upstream.internal','N') ;"
2026-09-26 20:54:31,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:54:32,127 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:54:32,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:54:32,362 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:54:32,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:54:32,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:54:32,846 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:54:32,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:54:33.776 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.792 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.807 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.821 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.837 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.853 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.908 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.914 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.929 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.939 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.949 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.962 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.971 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.978 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.984 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:33.990 UTC [21317] WARNING: there is no transaction in progress
2026-09-26 20:54:34,583 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:54:34,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:54:34,816 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:54:34,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:54:35,077 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:54:35,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:54:35,356 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:54:35,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:54:35,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:54:35,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2dg3.c.gcp-upstream.internal';"
2026-09-26 20:54:36,088 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:54:36,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:54:36,318 [I] DB_PATCHES have already been applied
I20260926 20:54:36.324221 9190 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpGD2b0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:54:36.324316 9190 mini_ranger.cc:193] Using host: 127.8.249.148
I20260926 20:54:36.339455 9190 mini_ranger.cc:227] Ranger admin URL: http://127.8.249.148:50367
Sep 26, 2026 8:54:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:54:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:54:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:54:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:54:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:54:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:54:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:54:37.474160 9190 mini_ranger.cc:160] Time spent starting Ranger: real 8.326s user 0.008s sys 0.033s
Sep 26, 2026 8:54:56 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:54:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 8:54:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:54:56 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:54:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:54:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:54:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:54:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:54:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:54:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:54:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:54:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:54:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:54:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:54:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:54:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:54:56 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:54:57 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:54:58.128742 9190 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/rangerkms_127.8.249.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/rangerkms_127.8.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.249.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.249.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.249.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/HTTP_127.8.249.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.249.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/HTTP_127.8.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:54:58 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456098, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:54:58.294126 9190 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:54:58.320793 9190 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:54:59.117523 9190 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:54:59.117638 9190 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-kms
I20260926 20:54:59.117686 9190 mini_ranger_kms.cc:227] Using postgres at 127.8.249.148:35511
2026-09-26 20:54:59,479 [I] DB FLAVOR :POSTGRES
2026-09-26 20:54:59,480 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:54:59,480 [I] Checking connection
2026-09-26 20:54:59,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:54:59,728 [I] connection success
2026-09-26 20:54:59,728 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:54:59,728 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:54:59,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:54:59,964 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:54:59,964 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:54:59,964 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:54:59,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.249.148:35511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:55:00,234 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:55:00.712152 9190 mini_ranger_kms.cc:310] Created kms service
I20260926 20:55:00.881490 9190 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:55:00.926812 9190 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:55:00 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456098, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:55:01.334256 9190 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:55:01.334632 9190 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpGD2b0/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpGD2b0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:55:01.334720 9190 mini_ranger_kms.cc:240] Using host: 127.8.249.148
I20260926 20:55:01.354200 9190 mini_ranger_kms.cc:276] Ranger KMS PID: 21497
I20260926 20:55:01.354362 9190 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.249.148:43927
Sep 26, 2026 8:55:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:55:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:55:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:55:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:55:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:55:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 20:55:03.633260 9190 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.339s user 0.088s sys 0.291s
I20260926 20:55:03.633517 9190 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 20:55:03.633699 9190 mini_ranger_kms.cc:401] 127.8.249.148:43927/kms/v1/keys
Sep 26 20:55:03 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.249.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 20:55:04 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.249.148@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
Sep 26 20:55:04 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456103, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.249.148@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
Sep 26 20:55:06 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:55:07.020371 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:41751
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:41751
--ranger_config_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.249.148:43927/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:07.400442 21548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:07.400789 21548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:07.400887 21548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:07.408510 21548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:07.408643 21548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:07.408697 21548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:07.408768 21548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:07.408833 21548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:07.419534 21548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.249.148:43927/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:41751
--ranger_config_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:41751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:07.420835 21548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:07.422150 21548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:07.434769 21556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:07.434742 21553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:07.434741 21554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:07.435251 21548 server_base.cc:1034] running on GCE node
I20260926 20:55:07.436313 21548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:07.437708 21548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:07.438895 21548 hybrid_clock.cc:648] HybridClock initialized: now 1790456107438837 us; error 33 us; skew 500 ppm
I20260926 20:55:07.439239 21548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:07 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456107, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:07.447209 21548 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:55:07.448729 21548 webserver.cc:460] Webserver started at http://127.8.249.190:34145/ using document root <none> and password file <none>
I20260926 20:55:07.449431 21548 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:07.449565 21548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:07.449867 21548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:55:07 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456107, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:55:07.500291 21548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "9b36c244ffdb45368ba0d35a6c9474c4"
format_stamp: "Formatted at 2026-09-26 20:55:07 on dist-test-slave-2dg3"
server_key: "039f3d9d795e1629a04f5201fea49102"
server_key_iv: "113b761fa2b244734f2cb6b67cd195be"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:07.501411 21548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "9b36c244ffdb45368ba0d35a6c9474c4"
format_stamp: "Formatted at 2026-09-26 20:55:07 on dist-test-slave-2dg3"
server_key: "039f3d9d795e1629a04f5201fea49102"
server_key_iv: "113b761fa2b244734f2cb6b67cd195be"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:07.507870 21548 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 20:55:07.529187 21563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:07.531025 21548 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 20:55:07.531234 21548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "9b36c244ffdb45368ba0d35a6c9474c4"
format_stamp: "Formatted at 2026-09-26 20:55:07 on dist-test-slave-2dg3"
server_key: "039f3d9d795e1629a04f5201fea49102"
server_key_iv: "113b761fa2b244734f2cb6b67cd195be"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:07.531482 21548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:07.557896 21548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:07.561801 21548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:07.562124 21548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:07.580821 21548 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:41751
I20260926 20:55:07.580861 21614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:41751 every 8 connection(s)
I20260926 20:55:07.582203 21548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:07.586544 21615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:07.589653 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 21548
I20260926 20:55:07.589872 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:55:07.590729 9190 external_mini_cluster.cc:1428] Setting key 29b517b753743c038a65782bd48ebb28
I20260926 20:55:07.599741 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4: Bootstrap starting.
I20260926 20:55:07.604025 21615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:07.605436 21615 log.cc:824] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:55:07 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:07.609332 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4: No bootstrap required, opened a new log
I20260926 20:55:07.615890 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:07.593003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.8.249.190:41751)
0926 20:55:07.593804 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:07.593816 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:07.593858 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:55:07.595559 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:07.595581 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:07.595974 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:07.596312 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:07.597911 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:07.599261 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:07.601044 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:07.601754 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:07.601896 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:07.606784 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:07.606850 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:07.606884 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:07.606934 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:07.610863 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:07.611588 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:07.611599 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:07.611613 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:07.611816 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:07.612170 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:07.612177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:07.612180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:07.612948 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:07.613295 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:07.613890 (+ 595us) server_negotiation.cc:299] Negotiation successful
0926 20:55:07.614337 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":402,"threads_started":1}
I20260926 20:55:07.619817 21615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41751 } }
I20260926 20:55:07.620155 21615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:07.620241 21615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36c244ffdb45368ba0d35a6c9474c4, State: Initialized, Role: FOLLOWER
I20260926 20:55:07.620713 21615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41751 } }
I20260926 20:55:07.621022 21615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:55:07.621167 21615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:55:07.621376 21615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:07.623550 21615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41751 } }
I20260926 20:55:07.624076 21615 leader_election.cc:302] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b36c244ffdb45368ba0d35a6c9474c4; no voters:
I20260926 20:55:07.624641 21615 leader_election.cc:288] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:55:07.624778 21620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:07.625456 21620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 1 LEADER]: Becoming Leader. State: Replica: 9b36c244ffdb45368ba0d35a6c9474c4, State: Running, Role: LEADER
I20260926 20:55:07.625960 21620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41751 } }
I20260926 20:55:07.626701 21615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:07.629542 21622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b36c244ffdb45368ba0d35a6c9474c4. Latest consensus state: current_term: 1 leader_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41751 } } }
I20260926 20:55:07.629566 21621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41751 } } }
I20260926 20:55:07.629987 21622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:07.630012 21621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:07.631879 21625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:07.638064 21625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:07.638666 21615 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:55:07.656037 21625 catalog_manager.cc:1347] Generated new cluster ID: 2bce4d33b990437cb3d3224c1e091042
I20260926 20:55:07.656198 21625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:55:07.689777 21625 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:55:07.691823 21625 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:07.700006 21625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4: Generated new TSK 0
I20260926 20:55:07.701125 21625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:55:08.398360 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.249.148:43927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:41751
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:55:08 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790456108, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 20:55:08.809703 21660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:08.810099 21660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:08.810271 21660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:08.821905 21660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:08.822059 21660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:08.822248 21660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:55:08.836292 21660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.249.148:43927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:41751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:08.837918 21660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:08.839663 21660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:08.855836 21670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:08.857436 21671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:08.868489 21673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:08.869071 21660 server_base.cc:1034] running on GCE node
I20260926 20:55:08.869881 21660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:08.871148 21660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:08.872464 21660 hybrid_clock.cc:648] HybridClock initialized: now 1790456108872388 us; error 67 us; skew 500 ppm
I20260926 20:55:08.872852 21660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:08 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456108, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:08.877202 21660 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:55:08.879338 21660 webserver.cc:460] Webserver started at http://127.8.249.129:43987/ using document root <none> and password file <none>
I20260926 20:55:08.880409 21660 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:08.880574 21660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:08.880999 21660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:55:08 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456108, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:55:08.904085 21660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "021fe34b876d440eb002b07f802abdd2"
format_stamp: "Formatted at 2026-09-26 20:55:08 on dist-test-slave-2dg3"
server_key: "28402c4f7669a5e15f3d8db27dd7dab2"
server_key_iv: "e975c9a9e1dce1fcdafabe8a68b4d53f"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:08.905344 21660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "021fe34b876d440eb002b07f802abdd2"
format_stamp: "Formatted at 2026-09-26 20:55:08 on dist-test-slave-2dg3"
server_key: "28402c4f7669a5e15f3d8db27dd7dab2"
server_key_iv: "e975c9a9e1dce1fcdafabe8a68b4d53f"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:08.929382 21660 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.008s sys 0.012s
I20260926 20:55:08.968374 21680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:08.971315 21660 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.015s sys 0.005s
I20260926 20:55:08.971700 21660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "021fe34b876d440eb002b07f802abdd2"
format_stamp: "Formatted at 2026-09-26 20:55:08 on dist-test-slave-2dg3"
server_key: "28402c4f7669a5e15f3d8db27dd7dab2"
server_key_iv: "e975c9a9e1dce1fcdafabe8a68b4d53f"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:08.971978 21660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:09.006057 21660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:09.016721 21660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:09.017073 21660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:09.023209 21660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:09.025157 21660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:09.025260 21660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:09.025398 21660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:09.025463 21660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:09.107267 21660 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:39731
I20260926 20:55:09.107446 21792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:39731 every 8 connection(s)
I20260926 20:55:09.108729 21660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:55:09.114259 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 21660
I20260926 20:55:09.114501 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:55:09.115686 9190 external_mini_cluster.cc:1428] Setting key 026a06655c438fcb7517a79857fdf098
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
Sep 26 20:55:09 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456108, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
I20260926 20:55:09.163460 21797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:09.127886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:54371 (local address 127.8.249.190:41751)
0926 20:55:09.135592 (+ 7706us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:09.135599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:09.136419 (+ 820us) server_negotiation.cc:407] Connection header received
0926 20:55:09.139201 (+ 2782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:09.139210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:09.139318 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:09.139424 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:09.141697 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:09.142540 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:09.145272 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:09.145999 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:09.146109 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:09.153156 (+ 7047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:09.153166 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:09.153174 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:09.153239 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:09.157248 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:09.158878 (+ 1630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:09.158887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:09.158893 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:09.159063 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:09.159671 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:09.159679 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:09.159685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:09.161337 (+ 1652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:09.161498 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:09.162645 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0926 20:55:09.162949 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7582,"thread_start_us":8019,"threads_started":1}
I20260926 20:55:09.166009 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:09.127197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:41751 (local address 127.8.249.129:54371)
0926 20:55:09.135794 (+ 8597us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:09.135844 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:09.138708 (+ 2864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:09.139883 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:09.139898 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:09.140578 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:09.141480 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:09.141509 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:09.142708 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:09.142712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:09.145095 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:09.145110 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:09.146489 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:09.146499 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:09.146831 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:09.147677 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:09.147714 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:09.151081 (+ 3367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:09.157449 (+ 6368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:09.157461 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:09.157484 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:09.158137 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:09.159230 (+ 1093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:09.159238 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:09.159245 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:09.159538 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:09.161596 (+ 2058us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:09.161607 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:09.162123 (+ 516us) client_negotiation.cc:769] Sending connection context
0926 20:55:09.163788 (+ 1665us) client_negotiation.cc:240] Negotiation successful
0926 20:55:09.164454 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7918,"thread_start_us":7646,"threads_started":1}
I20260926 20:55:09.168051 21793 heartbeater.cc:347] Connected to a master server at 127.8.249.190:41751
I20260926 20:55:09.168390 21793 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:09.169281 21793 heartbeater.cc:511] Master 127.8.249.190:41751 requested a full tablet report, sending...
I20260926 20:55:09.172024 21580 ts_manager.cc:194] Registered new tserver with Master: 021fe34b876d440eb002b07f802abdd2 (127.8.249.129:39731)
I20260926 20:55:09.174093 21580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:54371
Sep 26 20:55:09 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456108, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:55:09.201810 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.249.148:43927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:41751
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:55:09 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456108, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
W20260926 20:55:09.700747 21799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:09.701249 21799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:09.701448 21799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:09.712841 21799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:09.713086 21799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:09.713328 21799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:55:09.729627 21799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.249.148:43927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:41751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:09.732199 21799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:09.734373 21799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:09.771148 21807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:09.776037 21806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:09.776700 21809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:09.777446 21799 server_base.cc:1034] running on GCE node
I20260926 20:55:09.778367 21799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:09.779744 21799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:09.781000 21799 hybrid_clock.cc:648] HybridClock initialized: now 1790456109780935 us; error 48 us; skew 500 ppm
I20260926 20:55:09.781701 21799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:09 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456109, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:09.786478 21799 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:55:09.794858 21799 webserver.cc:460] Webserver started at http://127.8.249.130:44011/ using document root <none> and password file <none>
I20260926 20:55:09.796025 21799 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:09.796252 21799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:09.796698 21799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:55:09 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456109, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:55:09.816370 21799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "caf5aec696b640fbb77e594ff185bb90"
format_stamp: "Formatted at 2026-09-26 20:55:09 on dist-test-slave-2dg3"
server_key: "680a2bbb24587d4dcb0d6cca014d7ca9"
server_key_iv: "d8ab3fbb346265572bd2ce36fbc9e2f9"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:09.817335 21799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "caf5aec696b640fbb77e594ff185bb90"
format_stamp: "Formatted at 2026-09-26 20:55:09 on dist-test-slave-2dg3"
server_key: "680a2bbb24587d4dcb0d6cca014d7ca9"
server_key_iv: "d8ab3fbb346265572bd2ce36fbc9e2f9"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:09.823563 21799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260926 20:55:09.838116 21823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:09.839960 21799 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:55:09.840207 21799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "caf5aec696b640fbb77e594ff185bb90"
format_stamp: "Formatted at 2026-09-26 20:55:09 on dist-test-slave-2dg3"
server_key: "680a2bbb24587d4dcb0d6cca014d7ca9"
server_key_iv: "d8ab3fbb346265572bd2ce36fbc9e2f9"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:09.840569 21799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:09.872426 21799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:09.876933 21799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:09.877357 21799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:09.878479 21799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:09.880204 21799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:09.880285 21799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:09.880455 21799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:09.880520 21799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:09.916769 21799 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:46575
I20260926 20:55:09.916788 21935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:46575 every 8 connection(s)
I20260926 20:55:09.918017 21799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:55:09.926134 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 21799
I20260926 20:55:09.926311 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:55:09.926858 9190 external_mini_cluster.cc:1428] Setting key 422001910e725767e12746e02b675683
Sep 26 20:55:09 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456109, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:09.939888 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:09.920991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:55131 (local address 127.8.249.190:41751)
0926 20:55:09.921449 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:09.921453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:09.922503 (+ 1050us) server_negotiation.cc:407] Connection header received
0926 20:55:09.924355 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:09.924360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:09.924459 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:09.924573 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:09.925963 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:09.926985 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:09.929248 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:09.929649 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:09.929778 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:09.933125 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:09.933131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:09.933136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:09.933180 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:09.936418 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:09.937121 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:09.937128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:09.937132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:09.937315 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:09.937750 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:09.937755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:09.937759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:09.938163 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:09.938322 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:09.939271 (+ 949us) server_negotiation.cc:299] Negotiation successful
0926 20:55:09.939506 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":247,"threads_started":1}
I20260926 20:55:09.941092 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:09.921593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:41751 (local address 127.8.249.130:55131)
0926 20:55:09.922355 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:09.922392 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:09.924082 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:09.924814 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:09.924821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:09.925247 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:09.925801 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:09.925818 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:09.927184 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:09.927192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:09.929105 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:09.929131 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:09.929856 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:09.929864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:09.930116 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:09.930828 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:09.930862 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:09.932912 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:09.936558 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:09.936566 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:09.936583 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:09.937002 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:09.937418 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:09.937424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:09.937428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:09.937634 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:09.938401 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:09.938412 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:09.938890 (+ 478us) client_negotiation.cc:769] Sending connection context
0926 20:55:09.939177 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 20:55:09.939731 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":344,"threads_started":1}
I20260926 20:55:09.942585 21936 heartbeater.cc:347] Connected to a master server at 127.8.249.190:41751
I20260926 20:55:09.942962 21936 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:09.943678 21936 heartbeater.cc:511] Master 127.8.249.190:41751 requested a full tablet report, sending...
I20260926 20:55:09.945117 21580 ts_manager.cc:194] Registered new tserver with Master: caf5aec696b640fbb77e594ff185bb90 (127.8.249.130:46575)
I20260926 20:55:09.946058 21580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:55131
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:55:09.990602 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.249.148:43927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:41751
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:55:10.178193 21793 heartbeater.cc:503] Master 127.8.249.190:41751 was elected leader, sending a full tablet report...
W20260926 20:55:10.337788 21942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:10.338104 21942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:10.338265 21942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:10.345382 21942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:10.345520 21942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:10.345709 21942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:55:10.355048 21942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.249.148:43927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:41751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:10.356326 21942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:10.357617 21942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:10.371021 21947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:10.371065 21948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:10.370934 21950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:10.371701 21942 server_base.cc:1034] running on GCE node
I20260926 20:55:10.372295 21942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:10.373217 21942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:10.374429 21942 hybrid_clock.cc:648] HybridClock initialized: now 1790456110374387 us; error 44 us; skew 500 ppm
I20260926 20:55:10.374747 21942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:10 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456110, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:10.377847 21942 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:55:10.379348 21942 webserver.cc:460] Webserver started at http://127.8.249.131:40033/ using document root <none> and password file <none>
I20260926 20:55:10.379992 21942 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:10.380126 21942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:10.380412 21942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:55:10 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456110, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:55:10.397858 21942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "18273264ce314741a362a4ad3a135ca1"
format_stamp: "Formatted at 2026-09-26 20:55:10 on dist-test-slave-2dg3"
server_key: "63f3613608fabb8b48cfc84b71b8e313"
server_key_iv: "93632463085df3fc1cf344303df963fa"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:10.398687 21942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "18273264ce314741a362a4ad3a135ca1"
format_stamp: "Formatted at 2026-09-26 20:55:10 on dist-test-slave-2dg3"
server_key: "63f3613608fabb8b48cfc84b71b8e313"
server_key_iv: "93632463085df3fc1cf344303df963fa"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:10.404209 21942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 20:55:10.418962 21957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:10.420475 21942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:55:10.420658 21942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "18273264ce314741a362a4ad3a135ca1"
format_stamp: "Formatted at 2026-09-26 20:55:10 on dist-test-slave-2dg3"
server_key: "63f3613608fabb8b48cfc84b71b8e313"
server_key_iv: "93632463085df3fc1cf344303df963fa"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:10.420917 21942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:10.442276 21942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:10.445842 21942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:10.446135 21942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:10.447151 21942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:10.448459 21942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:10.448560 21942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:10.448698 21942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:10.448772 21942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:10.480464 21942 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:39525
I20260926 20:55:10.480499 22069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:39525 every 8 connection(s)
I20260926 20:55:10.481819 21942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:55:10.487673 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 21942
I20260926 20:55:10.487841 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:55:10.488270 9190 external_mini_cluster.cc:1428] Setting key 49d94b1c22d091a162e5e2615b92c939
Sep 26 20:55:10 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456110, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:10.499807 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:10.484248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:45129 (local address 127.8.249.190:41751)
0926 20:55:10.484748 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:10.484752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:10.485414 (+ 662us) server_negotiation.cc:407] Connection header received
0926 20:55:10.487233 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:10.487238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:10.487350 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:10.487461 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:10.489144 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:10.489934 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:10.491545 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:10.491950 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:10.492046 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:10.495056 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:10.495061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:10.495064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:10.495094 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:10.497416 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:10.498013 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:10.498017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:10.498019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:10.498130 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:10.498486 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:10.498489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:10.498491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:10.498725 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:10.498892 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:10.499423 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 20:55:10.499576 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":259,"threads_started":1}
I20260926 20:55:10.500736 22073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:10.484566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:41751 (local address 127.8.249.131:45129)
0926 20:55:10.485256 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:10.485299 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:10.486971 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:10.487704 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:10.487712 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:10.488233 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:10.488971 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:10.488995 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:10.490091 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:10.490096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:10.491409 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:10.491426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:10.492132 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:10.492136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:10.492325 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:10.492903 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:10.492923 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:10.494870 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:10.497537 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:10.497543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:10.497555 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:10.497920 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:10.498209 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:10.498211 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:10.498225 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:10.498401 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:10.498855 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:10.498861 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:10.499128 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 20:55:10.499345 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 20:55:10.499678 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":280,"threads_started":1}
I20260926 20:55:10.502283 22070 heartbeater.cc:347] Connected to a master server at 127.8.249.190:41751
I20260926 20:55:10.502602 22070 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:10.503448 22070 heartbeater.cc:511] Master 127.8.249.190:41751 requested a full tablet report, sending...
I20260926 20:55:10.504776 21580 ts_manager.cc:194] Registered new tserver with Master: 18273264ce314741a362a4ad3a135ca1 (127.8.249.131:39525)
I20260926 20:55:10.505585 21580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:45129
I20260926 20:55:10.515631 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:55:10.517309 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 21660
Sep 26 20:55:10 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:55:10.533429 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:10.518272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.8.249.129:39731)
0926 20:55:10.518961 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:10.518972 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:10.519004 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:55:10.519069 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:10.519076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:10.519337 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:10.519467 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:10.520874 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:10.521893 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:10.523295 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:10.523819 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:10.523919 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:10.528143 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:10.528154 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:10.528168 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:10.528200 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:10.530908 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:10.531582 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:10.531588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:10.531590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:10.531718 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:10.532069 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:10.532072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:10.532074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:10.532393 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:10.532613 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:10.532905 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 20:55:10.533151 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":328,"threads_started":1}
I20260926 20:55:10.534672 21788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:10.623253 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 21660
I20260926 20:55:10.639557 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 21799
Sep 26 20:55:10 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:55:10.655248 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:10.640567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.8.249.130:46575)
0926 20:55:10.641062 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:10.641070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:10.641093 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:55:10.641165 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:10.641171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:10.641381 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:10.641549 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:10.642881 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:10.643851 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:10.645206 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:10.645813 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:10.645952 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:10.650007 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:10.650018 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:10.650032 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:10.650070 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:10.652706 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:10.653325 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:10.653331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:10.653333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:10.653455 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:10.653754 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:10.653756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:10.653758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:10.654081 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:10.654318 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:10.654564 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 20:55:10.654759 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":252,"threads_started":1}
I20260926 20:55:10.656494 21931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:10.737637 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 21799
I20260926 20:55:10.757426 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 21942
Sep 26 20:55:10 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:55:10.773831 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:10.758451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.8.249.131:39525)
0926 20:55:10.759185 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:10.759192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:10.759220 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:55:10.759285 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:10.759292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:10.759520 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:10.759645 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:10.760869 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:10.761850 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:10.763247 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:10.763825 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:10.763925 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:10.768309 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:10.768323 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:10.768340 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:10.768388 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:10.771333 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:10.772055 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:10.772062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:10.772064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:10.772188 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:10.772488 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:10.772491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:10.772493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:10.772806 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:10.773037 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:10.773251 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 20:55:10.773445 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":435,"threads_started":1}
I20260926 20:55:10.775163 22065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:10.858188 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 21942
I20260926 20:55:10.873799 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 21548
I20260926 20:55:10.875099 21610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:10.980599 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 21548
I20260926 20:55:11.004204 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:41751
--webserver_interface=127.8.249.190
--webserver_port=34145
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:41751
--ranger_config_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.249.148:43927/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.249.148:43927/kms with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:11.329644 22082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:11.329916 22082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:11.330001 22082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:11.337203 22082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:11.337301 22082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:11.337409 22082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:11.337505 22082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:11.337582 22082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:11.347610 22082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.249.148:43927/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:41751
--ranger_config_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:41751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=34145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:11.348868 22082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:11.350195 22082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:11.361747 22088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:11.361776 22090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:11.361806 22087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:11.362481 22082 server_base.cc:1034] running on GCE node
I20260926 20:55:11.363302 22082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:11.364646 22082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:11.365875 22082 hybrid_clock.cc:648] HybridClock initialized: now 1790456111365810 us; error 51 us; skew 500 ppm
I20260926 20:55:11.366205 22082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:11 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456111, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:11.369280 22082 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:55:11.370625 22082 webserver.cc:460] Webserver started at http://127.8.249.190:34145/ using document root <none> and password file <none>
I20260926 20:55:11.371315 22082 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:11.371452 22082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:55:11 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456111, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:55:11.438795 22082 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 20:55:11.441921 22098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:11.443346 22082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 20:55:11.443534 22082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "9b36c244ffdb45368ba0d35a6c9474c4"
format_stamp: "Formatted at 2026-09-26 20:55:07 on dist-test-slave-2dg3"
server_key: "039f3d9d795e1629a04f5201fea49102"
server_key_iv: "113b761fa2b244734f2cb6b67cd195be"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:11.444471 22082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:11.475359 22082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:11.479588 22082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:11.479889 22082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:11.497130 22082 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:41751
I20260926 20:55:11.497151 22149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:41751 every 8 connection(s)
I20260926 20:55:11.498679 22082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:11.505525 22150 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:55:11.508034 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 22082
I20260926 20:55:11.508782 22150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4: Bootstrap starting.
I20260926 20:55:11.508873 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.249.148:43927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:39731
--local_ip_for_outbound_sockets=127.8.249.129
--tserver_master_addrs=127.8.249.190:41751
--webserver_port=43987
--webserver_interface=127.8.249.129
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.249.148:43927/kms with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:55:11.532191 22150 log.cc:824] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:11.543610 22150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:55:11.544203 22150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4: Bootstrap complete.
I20260926 20:55:11.555204 22150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41751 } }
I20260926 20:55:11.556192 22150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36c244ffdb45368ba0d35a6c9474c4, State: Initialized, Role: FOLLOWER
I20260926 20:55:11.556854 22150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41751 } }
I20260926 20:55:11.557097 22150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:55:11.557225 22150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:55:11.557410 22150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:55:11.561014 22150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41751 } }
I20260926 20:55:11.561482 22150 leader_election.cc:302] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b36c244ffdb45368ba0d35a6c9474c4; no voters:
I20260926 20:55:11.562126 22150 leader_election.cc:288] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:55:11.562264 22154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:55:11.563560 22154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [term 2 LEADER]: Becoming Leader. State: Replica: 9b36c244ffdb45368ba0d35a6c9474c4, State: Running, Role: LEADER
I20260926 20:55:11.564153 22154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41751 } }
I20260926 20:55:11.564345 22150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:11.566924 22155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41751 } } }
I20260926 20:55:11.567740 22155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:11.567636 22156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b36c244ffdb45368ba0d35a6c9474c4. Latest consensus state: current_term: 2 leader_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b36c244ffdb45368ba0d35a6c9474c4" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 41751 } } }
I20260926 20:55:11.568367 22156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:11.577907 22160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:11.583359 22160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:11.585089 22160 catalog_manager.cc:1259] Loaded cluster ID: 2bce4d33b990437cb3d3224c1e091042
I20260926 20:55:11.585284 22160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:55:11.590360 22160 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:11.592216 22160 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4: Loaded TSK: 0
I20260926 20:55:11.594228 22160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:55:11.601992 22150 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 20:55:11.970366 22152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:11.973755 22152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:11.973978 22152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:11.986012 22152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:11.986160 22152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:11.986343 22152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:55:12.002526 22152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.249.148:43927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:39731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=43987
--tserver_master_addrs=127.8.249.190:41751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:12.004367 22152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:12.006204 22152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:12.050685 22188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:12.051832 22187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:12.053562 22190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:12.064684 22152 server_base.cc:1034] running on GCE node
I20260926 20:55:12.065603 22152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:12.068943 22152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:12.070191 22152 hybrid_clock.cc:648] HybridClock initialized: now 1790456112070111 us; error 78 us; skew 500 ppm
I20260926 20:55:12.070626 22152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:12 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:12.075227 22152 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:55:12.088871 22152 webserver.cc:460] Webserver started at http://127.8.249.129:43987/ using document root <none> and password file <none>
I20260926 20:55:12.089987 22152 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:12.090179 22152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:55:12 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:55:12.164759 22152 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.006s sys 0.001s
I20260926 20:55:12.179559 22199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:12.181905 22152 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.001s sys 0.011s
I20260926 20:55:12.182130 22152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "021fe34b876d440eb002b07f802abdd2"
format_stamp: "Formatted at 2026-09-26 20:55:08 on dist-test-slave-2dg3"
server_key: "28402c4f7669a5e15f3d8db27dd7dab2"
server_key_iv: "e975c9a9e1dce1fcdafabe8a68b4d53f"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:12.183681 22152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:12.254189 22152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:12.259420 22152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:12.259769 22152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:12.261029 22152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:12.263211 22152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:12.263324 22152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:12.263427 22152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:12.263499 22152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:12.340860 22152 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:39731
I20260926 20:55:12.340932 22311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:39731 every 8 connection(s)
I20260926 20:55:12.342741 22152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:55:12.350863 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 22152
I20260926 20:55:12.351943 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.249.148:43927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:46575
--local_ip_for_outbound_sockets=127.8.249.130
--tserver_master_addrs=127.8.249.190:41751
--webserver_port=44011
--webserver_interface=127.8.249.130
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.249.148:43927/kms with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:55:12 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:12.392012 22314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:12.347031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:41751 (local address 127.8.249.129:34661)
0926 20:55:12.362159 (+ 15128us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:12.362244 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:12.365737 (+ 3493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:12.368334 (+ 2597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:12.368349 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:12.369024 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:12.369867 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:12.369897 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:12.372548 (+ 2651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:12.372556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:12.374955 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:12.374979 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:12.376780 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:12.376789 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:12.377072 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:12.379072 (+ 2000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:12.379114 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:12.382116 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:12.386998 (+ 4882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:12.387008 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:12.387026 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:12.387614 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:12.388126 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:12.388130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:12.388134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:12.388395 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:12.389506 (+ 1111us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:12.389515 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:12.389958 (+ 443us) client_negotiation.cc:769] Sending connection context
0926 20:55:12.390278 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 20:55:12.390908 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14743,"thread_start_us":16490,"threads_started":1}
I20260926 20:55:12.394613 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:12.347748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:34661 (local address 127.8.249.190:41751)
0926 20:55:12.358392 (+ 10644us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:12.358418 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:12.362422 (+ 4004us) server_negotiation.cc:407] Connection header received
0926 20:55:12.366556 (+ 4134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:12.366577 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:12.367154 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:12.367569 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:12.370189 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:12.372329 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:12.375208 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:12.376500 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:12.376752 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:12.382523 (+ 5771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:12.382534 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:12.382557 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:12.382616 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:12.386735 (+ 4119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:12.387803 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:12.387817 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:12.387825 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:12.387993 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:12.388525 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:12.388530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:12.388533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:12.389186 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:12.389403 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:12.392739 (+ 3336us) server_negotiation.cc:299] Negotiation successful
0926 20:55:12.393190 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10290,"thread_start_us":10017,"threads_started":1}
I20260926 20:55:12.398357 22312 heartbeater.cc:347] Connected to a master server at 127.8.249.190:41751
I20260926 20:55:12.398716 22312 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:12.399672 22312 heartbeater.cc:511] Master 127.8.249.190:41751 requested a full tablet report, sending...
W20260926 20:55:12.403049 22312 heartbeater.cc:644] Failed to heartbeat to 127.8.249.190:41751 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:55:12.403294 22312 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:12.403589 22312 heartbeater.cc:511] Master 127.8.249.190:41751 requested a full tablet report, sending...
I20260926 20:55:12.404793 22312 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:12.405236 22312 heartbeater.cc:511] Master 127.8.249.190:41751 requested a full tablet report, sending...
W20260926 20:55:12.406349 22312 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:55:12.779215 22316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:12.779565 22316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:12.779731 22316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:12.787415 22316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:12.787577 22316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:12.787768 22316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:55:12.798298 22316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.249.148:43927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:46575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=44011
--tserver_master_addrs=127.8.249.190:41751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:12.799834 22316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:12.801259 22316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:12.819947 22332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:12.820467 22333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:12.828361 22335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 26 20:55:12 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790456112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:12.831550 22316 server_base.cc:1034] running on GCE node
I20260926 20:55:12.832439 22316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:12.833786 22316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:12.835031 22316 hybrid_clock.cc:648] HybridClock initialized: now 1790456112834966 us; error 50 us; skew 500 ppm
I20260926 20:55:12.835477 22316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:12 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:12.842728 22316 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:55:12.856501 22316 webserver.cc:460] Webserver started at http://127.8.249.130:44011/ using document root <none> and password file <none>
I20260926 20:55:12.857609 22316 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:12.857776 22316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:55:12 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:55:12.890738 22316 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:55:12.894224 22343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:12.895924 22316 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:55:12.896133 22316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "caf5aec696b640fbb77e594ff185bb90"
format_stamp: "Formatted at 2026-09-26 20:55:09 on dist-test-slave-2dg3"
server_key: "680a2bbb24587d4dcb0d6cca014d7ca9"
server_key_iv: "d8ab3fbb346265572bd2ce36fbc9e2f9"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:12.897670 22316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:12.921567 22316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:12.927453 22316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:12.927822 22316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:12.929185 22316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:12.931074 22316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:12.931195 22316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:12.931313 22316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:12.931389 22316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:12.977731 22316 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:46575
I20260926 20:55:12.977772 22456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:46575 every 8 connection(s)
I20260926 20:55:12.979306 22316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:55:12.985723 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 22316
I20260926 20:55:12.986745 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.249.148:43927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:39525
--local_ip_for_outbound_sockets=127.8.249.131
--tserver_master_addrs=127.8.249.190:41751
--webserver_port=40033
--webserver_interface=127.8.249.131
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.249.148:43927/kms with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:55:13 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:13.043620 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:12.982468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:60819 (local address 127.8.249.190:41751)
0926 20:55:13.024304 (+ 41836us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:13.024311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:13.024333 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:55:13.024399 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:13.024407 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:13.024523 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:13.024648 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:13.026248 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:13.027061 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:13.028759 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:13.029173 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:13.029303 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:13.033241 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:13.033249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:13.033254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:13.033297 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:13.036862 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:13.040160 (+ 3298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:13.040166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:13.040170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:13.040377 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:13.040973 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:13.040978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:13.040982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:13.041384 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:13.041545 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:13.042968 (+ 1423us) server_negotiation.cc:299] Negotiation successful
0926 20:55:13.043214 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41691,"thread_start_us":41567,"threads_started":1}
I20260926 20:55:13.046330 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:12.983223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:41751 (local address 127.8.249.130:60819)
0926 20:55:13.019770 (+ 36547us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:13.019851 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:13.023377 (+ 3526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:13.024961 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:13.024970 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:13.025439 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:13.026055 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:13.026073 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:13.027223 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:13.027227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:13.028601 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:13.028620 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:13.029410 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:13.029419 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:13.029648 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:13.030346 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:13.030370 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:13.032940 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:13.037025 (+ 4085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:13.037034 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:13.037052 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:13.039996 (+ 2944us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:13.040543 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:13.040548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:13.040552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:13.040821 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:13.041627 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:13.041635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:13.042045 (+ 410us) client_negotiation.cc:769] Sending connection context
0926 20:55:13.042363 (+ 318us) client_negotiation.cc:240] Negotiation successful
0926 20:55:13.045047 (+ 2684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36057,"thread_start_us":35746,"threads_started":1}
I20260926 20:55:13.048164 22457 heartbeater.cc:347] Connected to a master server at 127.8.249.190:41751
I20260926 20:55:13.048517 22457 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:13.049418 22457 heartbeater.cc:511] Master 127.8.249.190:41751 requested a full tablet report, sending...
I20260926 20:55:13.051517 22115 ts_manager.cc:194] Registered new tserver with Master: caf5aec696b640fbb77e594ff185bb90 (127.8.249.130:46575)
I20260926 20:55:13.053711 22115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:60819
Sep 26 20:55:13 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:55:13.406522 22312 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:13.407016 22312 heartbeater.cc:511] Master 127.8.249.190:41751 requested a full tablet report, sending...
I20260926 20:55:13.408310 22115 ts_manager.cc:194] Registered new tserver with Master: 021fe34b876d440eb002b07f802abdd2 (127.8.249.129:39731)
I20260926 20:55:13.409690 22115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:34661
W20260926 20:55:13.444044 22461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:13.444404 22461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:13.444540 22461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:13.455783 22461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:13.455921 22461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:13.456090 22461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:55:13.471392 22461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.249.148:43927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:39525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=40033
--tserver_master_addrs=127.8.249.190:41751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:13.473039 22461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:13.474713 22461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:13.491362 22466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:13.493188 22467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:13.512508 22469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:13.512809 22461 server_base.cc:1034] running on GCE node
I20260926 20:55:13.513645 22461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:13.514971 22461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:13.516229 22461 hybrid_clock.cc:648] HybridClock initialized: now 1790456113516156 us; error 57 us; skew 500 ppm
I20260926 20:55:13.516598 22461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:13 dist-test-slave-2dg3 krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456113, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:13.520880 22461 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:55:13.525229 22461 webserver.cc:460] Webserver started at http://127.8.249.131:40033/ using document root <none> and password file <none>
I20260926 20:55:13.526263 22461 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:13.526482 22461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:55:13 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456113, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:55:13.555743 22461 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:55:13.559798 22478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:13.562309 22461 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260926 20:55:13.562502 22461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "18273264ce314741a362a4ad3a135ca1"
format_stamp: "Formatted at 2026-09-26 20:55:10 on dist-test-slave-2dg3"
server_key: "63f3613608fabb8b48cfc84b71b8e313"
server_key_iv: "93632463085df3fc1cf344303df963fa"
server_key_version: "kuduclusterkey@0"
I20260926 20:55:13.563913 22461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 26 20:55:13 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for HTTP/127.8.249.148@KRBTEST.COM
I20260926 20:55:13.590984 22461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:13.595906 22461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:13.596197 22461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:13.597436 22461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:13.599046 22461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:13.599151 22461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:13.599261 22461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:13.599334 22461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:13.940907 22461 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:39525
I20260926 20:55:13.940973 22597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:39525 every 8 connection(s)
I20260926 20:55:13.942724 22461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:55:13.953204 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 22461
Sep 26 20:55:13 dist-test-slave-2dg3 krb5kdc[21104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456113, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:13.967813 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:13.945355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:53647 (local address 127.8.249.190:41751)
0926 20:55:13.945807 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:13.945811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:13.946893 (+ 1082us) server_negotiation.cc:407] Connection header received
0926 20:55:13.949003 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:13.949009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:13.949110 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:13.949221 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:13.950647 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:13.951823 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:13.953835 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:13.954532 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:13.954652 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:13.959494 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:13.959502 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:13.959507 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:13.959582 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:13.963396 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:13.964574 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:13.964581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:13.964584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:13.964757 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:13.965400 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:13.965406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:13.965410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:13.965843 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:13.965970 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:13.967130 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0926 20:55:13.967358 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":243,"threads_started":1}
I20260926 20:55:13.969414 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:13.945995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:41751 (local address 127.8.249.131:53647)
0926 20:55:13.946718 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:13.946761 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:13.948691 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:13.949465 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:13.949474 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:13.949908 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:13.950456 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:13.950472 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:13.951971 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:13.951975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:13.953602 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:13.953626 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:13.954995 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:13.955001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:13.955263 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:13.956157 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:13.956200 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:13.959161 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:13.963748 (+ 4587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:13.963760 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:13.963779 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:13.964382 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:13.964972 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:13.964977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:13.964981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:13.965257 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:13.966086 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:13.966095 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:13.966626 (+ 531us) client_negotiation.cc:769] Sending connection context
0926 20:55:13.967595 (+ 969us) client_negotiation.cc:240] Negotiation successful
0926 20:55:13.968065 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":302,"threads_started":1}
I20260926 20:55:13.970980 22598 heartbeater.cc:347] Connected to a master server at 127.8.249.190:41751
I20260926 20:55:13.971290 22598 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:13.972002 22598 heartbeater.cc:511] Master 127.8.249.190:41751 requested a full tablet report, sending...
I20260926 20:55:13.973593 22115 ts_manager.cc:194] Registered new tserver with Master: 18273264ce314741a362a4ad3a135ca1 (127.8.249.131:39525)
I20260926 20:55:13.974480 22115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:53647
I20260926 20:55:13.987254 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:13.954334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.8.249.190:41751)
0926 20:55:13.967202 (+ 12868us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:13.967208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:13.967503 (+ 295us) server_negotiation.cc:407] Connection header received
0926 20:55:13.968452 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:13.968458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:13.968546 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:13.968658 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:13.972758 (+ 4100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:13.973971 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:13.976064 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:13.976651 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:13.976770 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:13.981600 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:13.981605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:13.981609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:13.981641 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:13.984321 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:13.985019 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:13.985024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:13.985028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:13.985157 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:13.985529 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:13.985534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:13.985537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:13.985810 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:13.986102 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:13.986402 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 20:55:13.986642 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12735,"thread_start_us":12574,"threads_started":1}
I20260926 20:55:13.990358 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:55:14.009208 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:13.993523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.8.249.190:41751)
0926 20:55:13.993774 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:13.993783 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:13.994180 (+ 397us) server_negotiation.cc:407] Connection header received
0926 20:55:13.994362 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:13.994366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:13.994425 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:13.994504 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:13.996758 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:13.997417 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:13.999485 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:13.999975 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.000091 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.003725 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:14.003730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:14.003733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:14.003764 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:14.006391 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:14.007334 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:14.007338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:14.007340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:14.007469 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:14.007898 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:14.007901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:14.007903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:14.008167 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:14.008363 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:14.008753 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 20:55:14.008931 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:55:14.014369 22115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:55:14.021948 22115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:55:14.058702 22457 heartbeater.cc:503] Master 127.8.249.190:41751 was elected leader, sending a full tablet report...
I20260926 20:55:14.064479 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.038063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:39525 (local address 127.0.0.1:49090)
0926 20:55:14.054836 (+ 16773us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:14.054863 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:14.055038 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:14.056121 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:14.056127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:14.056152 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:14.056594 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:14.056607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.058660 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.058665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:14.061175 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:14.061190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.063430 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.063436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:14.063650 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.063703 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 20:55:14.063844 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 20:55:14.064032 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16572,"mutex_wait_us":261,"thread_start_us":22659,"threads_started":1}
I20260926 20:55:14.065438 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.039384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.8.249.131:39525)
0926 20:55:14.048035 (+ 8651us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:14.048045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:14.054981 (+ 6936us) server_negotiation.cc:407] Connection header received
0926 20:55:14.055250 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:14.055261 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:14.055560 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:14.055802 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:14.056816 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.058478 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.061348 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.062897 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.063006 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.063240 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:14.064927 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0926 20:55:14.065115 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8476,"thread_start_us":8331,"threads_started":1}
I20260926 20:55:14.065871 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.038902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.8.249.130:46575)
0926 20:55:14.047527 (+ 8625us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:14.047541 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:14.049787 (+ 2246us) server_negotiation.cc:407] Connection header received
0926 20:55:14.049915 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:14.049924 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:14.050242 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:14.050464 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:14.051365 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.054751 (+ 3386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.061380 (+ 6629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.063035 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.064927 (+ 1892us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.065165 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:14.065270 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 20:55:14.065447 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8435,"thread_start_us":8303,"threads_started":1}
I20260926 20:55:14.065677 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.038033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:46575 (local address 127.0.0.1:37852)
0926 20:55:14.048701 (+ 10668us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:14.048755 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:14.049007 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:14.050483 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:14.050492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:14.050527 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:14.051091 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:14.051107 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.054963 (+ 3856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.054971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:14.061150 (+ 6179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:14.061171 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.064562 (+ 3391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.064569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:14.065042 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.065072 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 20:55:14.065124 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 20:55:14.065318 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10348,"mutex_wait_us":279,"thread_start_us":17153,"threads_started":1}
I20260926 20:55:14.066450 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.040097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.8.249.129:39731)
0926 20:55:14.049703 (+ 9606us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:14.049714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:14.055956 (+ 6242us) server_negotiation.cc:407] Connection header received
0926 20:55:14.056303 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:14.056312 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:14.056598 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:14.056778 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:14.057706 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.059768 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.062625 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.064047 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.064218 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.064429 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:14.065831 (+ 1402us) server_negotiation.cc:299] Negotiation successful
0926 20:55:14.066000 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9408,"thread_start_us":10073,"threads_started":1}
I20260926 20:55:14.067178 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.039494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39731 (local address 127.0.0.1:40558)
0926 20:55:14.055830 (+ 16336us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:14.055850 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:14.056066 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:14.057121 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:14.057125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:14.057162 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:14.057499 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:14.057511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.059944 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.059949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:14.062451 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:14.062464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.064647 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.064655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:14.064890 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.064924 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:55:14.066685 (+ 1761us) client_negotiation.cc:240] Negotiation successful
0926 20:55:14.066847 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16206,"mutex_wait_us":15,"thread_start_us":8741,"threads_started":1}
I20260926 20:55:14.070305 22392 tablet_service.cc:1467] Processing CreateTablet for tablet 1c59dcba029a45ac893af1d0c5531754 (DEFAULT_TABLE table=test-table [id=5d4193fd6926471c8410011d14d26505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:14.070302 22526 tablet_service.cc:1467] Processing CreateTablet for tablet 1c59dcba029a45ac893af1d0c5531754 (DEFAULT_TABLE table=test-table [id=5d4193fd6926471c8410011d14d26505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:14.070506 22247 tablet_service.cc:1467] Processing CreateTablet for tablet 1c59dcba029a45ac893af1d0c5531754 (DEFAULT_TABLE table=test-table [id=5d4193fd6926471c8410011d14d26505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:14.071825 22392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c59dcba029a45ac893af1d0c5531754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:14.071823 22247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c59dcba029a45ac893af1d0c5531754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:14.071823 22526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c59dcba029a45ac893af1d0c5531754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:14.099745 22620 tablet_bootstrap.cc:492] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90: Bootstrap starting.
I20260926 20:55:14.100800 22619 tablet_bootstrap.cc:492] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2: Bootstrap starting.
I20260926 20:55:14.105204 22620 tablet_bootstrap.cc:654] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:14.105423 22619 tablet_bootstrap.cc:654] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:14.106045 22621 tablet_bootstrap.cc:492] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1: Bootstrap starting.
I20260926 20:55:14.106974 22619 log.cc:824] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:14.108403 22620 log.cc:824] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:14.109747 22621 tablet_bootstrap.cc:654] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:14.111059 22621 log.cc:824] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:14.117164 22619 tablet_bootstrap.cc:492] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2: No bootstrap required, opened a new log
I20260926 20:55:14.117658 22619 ts_tablet_manager.cc:1397] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20260926 20:55:14.125640 22620 tablet_bootstrap.cc:492] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90: No bootstrap required, opened a new log
I20260926 20:55:14.126000 22620 ts_tablet_manager.cc:1397] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.012s
I20260926 20:55:14.126159 22621 tablet_bootstrap.cc:492] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1: No bootstrap required, opened a new log
I20260926 20:55:14.126509 22621 ts_tablet_manager.cc:1397] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.016s
I20260926 20:55:14.134315 22619 raft_consensus.cc:348] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5aec696b640fbb77e594ff185bb90" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46575 } } peers { permanent_uuid: "18273264ce314741a362a4ad3a135ca1" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39525 } } peers { permanent_uuid: "021fe34b876d440eb002b07f802abdd2" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39731 } }
I20260926 20:55:14.135154 22619 raft_consensus.cc:374] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:14.135329 22619 raft_consensus.cc:729] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021fe34b876d440eb002b07f802abdd2, State: Initialized, Role: FOLLOWER
I20260926 20:55:14.136063 22619 consensus_queue.cc:260] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5aec696b640fbb77e594ff185bb90" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46575 } } peers { permanent_uuid: "18273264ce314741a362a4ad3a135ca1" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39525 } } peers { permanent_uuid: "021fe34b876d440eb002b07f802abdd2" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39731 } }
I20260926 20:55:14.136376 22620 raft_consensus.cc:348] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5aec696b640fbb77e594ff185bb90" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46575 } } peers { permanent_uuid: "18273264ce314741a362a4ad3a135ca1" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39525 } } peers { permanent_uuid: "021fe34b876d440eb002b07f802abdd2" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39731 } }
I20260926 20:55:14.136895 22620 raft_consensus.cc:374] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:14.137023 22620 raft_consensus.cc:729] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf5aec696b640fbb77e594ff185bb90, State: Initialized, Role: FOLLOWER
I20260926 20:55:14.137601 22621 raft_consensus.cc:348] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5aec696b640fbb77e594ff185bb90" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46575 } } peers { permanent_uuid: "18273264ce314741a362a4ad3a135ca1" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39525 } } peers { permanent_uuid: "021fe34b876d440eb002b07f802abdd2" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39731 } }
I20260926 20:55:14.138090 22621 raft_consensus.cc:374] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:14.137928 22620 consensus_queue.cc:260] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5aec696b640fbb77e594ff185bb90" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46575 } } peers { permanent_uuid: "18273264ce314741a362a4ad3a135ca1" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39525 } } peers { permanent_uuid: "021fe34b876d440eb002b07f802abdd2" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39731 } }
I20260926 20:55:14.138233 22621 raft_consensus.cc:729] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18273264ce314741a362a4ad3a135ca1, State: Initialized, Role: FOLLOWER
I20260926 20:55:14.138971 22621 consensus_queue.cc:260] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5aec696b640fbb77e594ff185bb90" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46575 } } peers { permanent_uuid: "18273264ce314741a362a4ad3a135ca1" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39525 } } peers { permanent_uuid: "021fe34b876d440eb002b07f802abdd2" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39731 } }
I20260926 20:55:14.143867 22620 ts_tablet_manager.cc:1428] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90: Time spent starting tablet: real 0.018s user 0.012s sys 0.003s
I20260926 20:55:14.160394 22312 heartbeater.cc:503] Master 127.8.249.190:41751 was elected leader, sending a full tablet report...
I20260926 20:55:14.161367 22619 ts_tablet_manager.cc:1428] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2: Time spent starting tablet: real 0.043s user 0.023s sys 0.014s
I20260926 20:55:14.161847 22625 raft_consensus.cc:482] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:55:14.162171 22625 raft_consensus.cc:504] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5aec696b640fbb77e594ff185bb90" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46575 } } peers { permanent_uuid: "18273264ce314741a362a4ad3a135ca1" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39525 } } peers { permanent_uuid: "021fe34b876d440eb002b07f802abdd2" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39731 } }
I20260926 20:55:14.171047 22598 heartbeater.cc:503] Master 127.8.249.190:41751 was elected leader, sending a full tablet report...
I20260926 20:55:14.171149 22625 leader_election.cc:288] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caf5aec696b640fbb77e594ff185bb90 (127.8.249.130:46575), 18273264ce314741a362a4ad3a135ca1 (127.8.249.131:39525)
I20260926 20:55:14.172771 22621 ts_tablet_manager.cc:1428] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1: Time spent starting tablet: real 0.046s user 0.013s sys 0.014s
I20260926 20:55:14.181023 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.173657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:44783 (local address 127.8.249.131:39525)
0926 20:55:14.173847 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:14.173854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:14.174354 (+ 500us) server_negotiation.cc:407] Connection header received
0926 20:55:14.174609 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:14.174614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:14.174680 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:14.174754 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:14.175613 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.176640 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.179042 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.180016 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.180119 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.180170 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:14.180640 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 20:55:14.180769 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:55:14.181304 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.173147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:39525 (local address 127.8.249.129:44783)
0926 20:55:14.174257 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:14.174275 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:14.174462 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:14.175008 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:14.175018 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:14.175041 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:14.175422 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:14.175435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.176841 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.176846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:14.178867 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:14.178882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.180365 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.180369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:14.180482 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.180503 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:55:14.180620 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:55:14.180764 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":43,"thread_start_us":4513,"threads_started":1}
I20260926 20:55:14.181417 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.164761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:36883 (local address 127.8.249.130:46575)
0926 20:55:14.164987 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:14.164994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:14.173867 (+ 8873us) server_negotiation.cc:407] Connection header received
0926 20:55:14.174213 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:14.174219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:14.174309 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:14.174375 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:14.175190 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.176273 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.179055 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.180292 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.180419 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.180486 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:14.181039 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 20:55:14.181188 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 20:55:14.182201 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.164633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:46575 (local address 127.8.249.129:36883)
0926 20:55:14.173743 (+ 9110us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:14.173765 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:14.173997 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:14.174570 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:14.174575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:14.174600 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:14.174979 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:14.174992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.176453 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.176458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:14.178874 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:14.178889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.180679 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.180685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:14.180857 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.180892 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:55:14.181765 (+ 873us) client_negotiation.cc:240] Negotiation successful
0926 20:55:14.181904 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8972,"mutex_wait_us":35,"thread_start_us":8732,"threads_started":1}
I20260926 20:55:14.182484 22553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c59dcba029a45ac893af1d0c5531754" candidate_uuid: "021fe34b876d440eb002b07f802abdd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18273264ce314741a362a4ad3a135ca1" is_pre_election: true
I20260926 20:55:14.183103 22553 raft_consensus.cc:2454] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021fe34b876d440eb002b07f802abdd2 in term 0.
I20260926 20:55:14.183121 22412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c59dcba029a45ac893af1d0c5531754" candidate_uuid: "021fe34b876d440eb002b07f802abdd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf5aec696b640fbb77e594ff185bb90" is_pre_election: true
I20260926 20:55:14.183588 22412 raft_consensus.cc:2454] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021fe34b876d440eb002b07f802abdd2 in term 0.
I20260926 20:55:14.183832 22200 leader_election.cc:302] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021fe34b876d440eb002b07f802abdd2, 18273264ce314741a362a4ad3a135ca1; no voters:
I20260926 20:55:14.184307 22628 raft_consensus.cc:2781] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:55:14.184450 22628 raft_consensus.cc:482] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:55:14.184603 22628 raft_consensus.cc:3037] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:14.186887 22628 raft_consensus.cc:504] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5aec696b640fbb77e594ff185bb90" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46575 } } peers { permanent_uuid: "18273264ce314741a362a4ad3a135ca1" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39525 } } peers { permanent_uuid: "021fe34b876d440eb002b07f802abdd2" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39731 } }
I20260926 20:55:14.187616 22628 leader_election.cc:288] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [CANDIDATE]: Term 1 election: Requested vote from peers caf5aec696b640fbb77e594ff185bb90 (127.8.249.130:46575), 18273264ce314741a362a4ad3a135ca1 (127.8.249.131:39525)
I20260926 20:55:14.188058 22412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c59dcba029a45ac893af1d0c5531754" candidate_uuid: "021fe34b876d440eb002b07f802abdd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf5aec696b640fbb77e594ff185bb90"
I20260926 20:55:14.188190 22553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c59dcba029a45ac893af1d0c5531754" candidate_uuid: "021fe34b876d440eb002b07f802abdd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18273264ce314741a362a4ad3a135ca1"
I20260926 20:55:14.188287 22412 raft_consensus.cc:3037] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:14.188397 22553 raft_consensus.cc:3037] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:14.191674 22412 raft_consensus.cc:2454] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021fe34b876d440eb002b07f802abdd2 in term 1.
I20260926 20:55:14.191674 22553 raft_consensus.cc:2454] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021fe34b876d440eb002b07f802abdd2 in term 1.
I20260926 20:55:14.192239 22200 leader_election.cc:302] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021fe34b876d440eb002b07f802abdd2, 18273264ce314741a362a4ad3a135ca1; no voters:
I20260926 20:55:14.192544 22628 raft_consensus.cc:2781] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:14.192727 22628 raft_consensus.cc:686] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [term 1 LEADER]: Becoming Leader. State: Replica: 021fe34b876d440eb002b07f802abdd2, State: Running, Role: LEADER
I20260926 20:55:14.193233 22628 consensus_queue.cc:237] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5aec696b640fbb77e594ff185bb90" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46575 } } peers { permanent_uuid: "18273264ce314741a362a4ad3a135ca1" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39525 } } peers { permanent_uuid: "021fe34b876d440eb002b07f802abdd2" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39731 } }
W20260926 20:55:14.194391 22599 tablet.cc:2369] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:14.198346 22115 catalog_manager.cc:5495] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 021fe34b876d440eb002b07f802abdd2 (127.8.249.129). New cstate: current_term: 1 leader_uuid: "021fe34b876d440eb002b07f802abdd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf5aec696b640fbb77e594ff185bb90" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18273264ce314741a362a4ad3a135ca1" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021fe34b876d440eb002b07f802abdd2" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39731 } health_report { overall_health: HEALTHY } } }
I20260926 20:55:14.223421 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.217196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.8.249.129:39731)
0926 20:55:14.217404 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:14.217408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:14.217427 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:55:14.217650 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:14.217657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:14.217768 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:14.217930 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:14.218269 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.219429 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.220850 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.221484 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.221615 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.222102 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:14.222838 (+ 736us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:14.222991 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:14.223041 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 20:55:14.223126 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:55:14.233294 22553 raft_consensus.cc:1261] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1 [term 1 FOLLOWER]: Refusing update from remote peer 021fe34b876d440eb002b07f802abdd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:14.233295 22412 raft_consensus.cc:1261] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90 [term 1 FOLLOWER]: Refusing update from remote peer 021fe34b876d440eb002b07f802abdd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 20:55:14.233301 22458 tablet.cc:2369] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:14.234683 22628 consensus_queue.cc:1035] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18273264ce314741a362a4ad3a135ca1" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:14.235141 22625 consensus_queue.cc:1035] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "caf5aec696b640fbb77e594ff185bb90" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:14.252066 22633 mvcc.cc:205] Tried to move back new op lower bound from 7333708243888590848 to 7333708243739574272. Current Snapshot: MvccSnapshot[applied={T|T < 7333708243888590848}]
I20260926 20:55:14.253759 22635 mvcc.cc:205] Tried to move back new op lower bound from 7333708243888590848 to 7333708243739574272. Current Snapshot: MvccSnapshot[applied={T|T < 7333708243888590848}]
I20260926 20:55:14.264657 22634 mvcc.cc:205] Tried to move back new op lower bound from 7333708243888590848 to 7333708243739574272. Current Snapshot: MvccSnapshot[applied={T|T < 7333708243888590848}]
I20260926 20:55:14.280990 22115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:55:14.281296 22115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:55:14.285652 22115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b36c244ffdb45368ba0d35a6c9474c4: Sending DeleteTablet for 3 replicas of tablet 1c59dcba029a45ac893af1d0c5531754
I20260926 20:55:14.286942 22526 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c59dcba029a45ac893af1d0c5531754 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:14 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:49090
I20260926 20:55:14.287328 22247 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c59dcba029a45ac893af1d0c5531754 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:14 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:40558
I20260926 20:55:14.287578 22392 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c59dcba029a45ac893af1d0c5531754 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:14 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:37852
I20260926 20:55:14.296002 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 22152
I20260926 20:55:14.305624 22643 tablet_replica.cc:330] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2: stopping tablet replica
I20260926 20:55:14.306099 22643 raft_consensus.cc:2229] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:55:14.306725 22643 raft_consensus.cc:2258] T 1c59dcba029a45ac893af1d0c5531754 P 021fe34b876d440eb002b07f802abdd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:14.311383 22644 tablet_replica.cc:330] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90: stopping tablet replica
I20260926 20:55:14.311864 22644 raft_consensus.cc:2229] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:55:14.312388 22644 raft_consensus.cc:2258] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:14.314471 22644 ts_tablet_manager.cc:1905] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:14.316763 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.296992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.8.249.129:39731)
0926 20:55:14.297174 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:14.297180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:14.297279 (+ 99us) server_negotiation.cc:407] Connection header received
0926 20:55:14.297475 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:14.297483 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:14.297577 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:14.297691 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:14.300220 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.301896 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.303739 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.304647 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.304753 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.308664 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:14.308677 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:14.308694 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:14.308739 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:14.312682 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:14.314095 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:14.314104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:14.314107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:14.314301 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:14.314764 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:14.314769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:14.314771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:14.315261 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:14.315574 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:14.316150 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 20:55:14.316394 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:55:14.318198 22307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:14.318430 22642 tablet_replica.cc:330] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1: stopping tablet replica
I20260926 20:55:14.318862 22642 raft_consensus.cc:2229] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:55:14.319353 22642 raft_consensus.cc:2258] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:14.321600 22642 ts_tablet_manager.cc:1905] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:14.327836 22644 ts_tablet_manager.cc:1918] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:14.328133 22644 log.cc:1196] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/1c59dcba029a45ac893af1d0c5531754
I20260926 20:55:14.328192 22642 ts_tablet_manager.cc:1918] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:14.328344 22642 log.cc:1196] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/1c59dcba029a45ac893af1d0c5531754
I20260926 20:55:14.328832 22642 ts_tablet_manager.cc:1939] T 1c59dcba029a45ac893af1d0c5531754 P 18273264ce314741a362a4ad3a135ca1: Deleting consensus metadata
I20260926 20:55:14.329035 22644 ts_tablet_manager.cc:1939] T 1c59dcba029a45ac893af1d0c5531754 P caf5aec696b640fbb77e594ff185bb90: Deleting consensus metadata
I20260926 20:55:14.330819 22099 catalog_manager.cc:4841] TS 18273264ce314741a362a4ad3a135ca1 (127.8.249.131:39525): tablet 1c59dcba029a45ac893af1d0c5531754 (table test-table [id=5d4193fd6926471c8410011d14d26505]) successfully deleted
I20260926 20:55:14.331379 22100 catalog_manager.cc:4841] TS caf5aec696b640fbb77e594ff185bb90 (127.8.249.130:46575): tablet 1c59dcba029a45ac893af1d0c5531754 (table test-table [id=5d4193fd6926471c8410011d14d26505]) successfully deleted
I20260926 20:55:14.419550 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 22152
I20260926 20:55:14.437275 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 22316
W20260926 20:55:14.437444 22101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.249.129:39731 (error 108)
I20260926 20:55:14.439424 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.438930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39731 (local address 127.0.0.1:40572)
0926 20:55:14.439105 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:14.439228 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 20:55:14.439670 22101 catalog_manager.cc:4584] TS 021fe34b876d440eb002b07f802abdd2 (127.8.249.129:39731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c59dcba029a45ac893af1d0c5531754: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39731: connect: Connection refused (error 111)
I20260926 20:55:14.451275 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.438533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.8.249.130:46575)
0926 20:55:14.438712 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:14.438717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:14.438737 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:55:14.438801 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:14.438833 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:14.438915 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:14.439013 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:14.440455 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.441186 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.442563 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.443134 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.443231 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.445682 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:14.445694 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:14.445710 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:14.445753 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:14.448528 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:14.449235 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:14.449243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:14.449245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:14.449370 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:14.449718 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:14.449724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:14.449727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:14.450052 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:14.450374 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:14.450662 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 20:55:14.450902 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:55:14.451929 22452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:14.493369 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.492897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39731 (local address 127.0.0.1:40584)
0926 20:55:14.493115 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:14.493232 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 20:55:14.494434 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.494098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39731 (local address 127.0.0.1:40592)
0926 20:55:14.494252 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:14.494328 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 20:55:14.541857 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 22316
I20260926 20:55:14.555014 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.554487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39731 (local address 127.0.0.1:40604)
0926 20:55:14.554680 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:14.554774 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 20:55:14.556061 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.555736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39731 (local address 127.0.0.1:40606)
0926 20:55:14.555878 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:14.555954 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:55:14.558894 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 22461
I20260926 20:55:14.572450 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.559584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.8.249.131:39525)
0926 20:55:14.559740 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:14.559744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:14.559762 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:55:14.559898 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:14.559902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:14.559962 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:14.560043 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:14.561320 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.562035 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.563352 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:14.563746 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:14.563845 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:14.566667 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:14.566682 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:14.566696 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:14.566779 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:14.569586 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:14.570289 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:14.570299 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:14.570303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:14.570436 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:14.570873 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:14.570879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:14.570882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:14.571265 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:14.571569 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:14.571882 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 20:55:14.572096 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:55:14.573132 22593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:14.636914 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.636425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39731 (local address 127.0.0.1:40616)
0926 20:55:14.636644 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:14.636743 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 20:55:14.638006 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:14.637608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39731 (local address 127.0.0.1:40628)
0926 20:55:14.637824 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:14.637894 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 20:55:14.662518 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 22461
I20260926 20:55:14.679098 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 22082
I20260926 20:55:14.680243 22145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:14.793167 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 22082
I20260926 20:55:14.815021 9190 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:55:15.151058 9190 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:55:15.151294 9190 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:55:15.496383 9190 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:55:15.496 UTC [21134] LOG: received smart shutdown request
2026-09-26 20:55:15.500 UTC [21134] LOG: background worker "logical replication launcher" (PID 21143) exited with exit code 1
2026-09-26 20:55:15.500 UTC [21138] LOG: shutting down
2026-09-26 20:55:15.640 UTC [21134] LOG: database system is shut down
2026-09-26T20:55:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51487 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:55:15 dist-test-slave-2dg3 krb5kdc[22658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:55:15 dist-test-slave-2dg3 krb5kdc[22658](info): set up 2 sockets
Sep 26 20:55:15 dist-test-slave-2dg3 krb5kdc[22658](info): commencing operation
krb5kdc: starting...
W20260926 20:55:16.875119 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:55:17 dist-test-slave-2dg3 krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:55:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:55:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:55:17.113943 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:43033
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:43033,127.8.249.189:46371,127.8.249.188:34391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:17.451743 22671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:17.452036 22671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:17.452158 22671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:17.459357 22671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:17.459473 22671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:17.459519 22671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:17.459568 22671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:17.459611 22671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:17.469645 22671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:43033,127.8.249.189:46371,127.8.249.188:34391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:43033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:17.471006 22671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:17.472273 22671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:17.484010 22679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:17.484088 22676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:17.484118 22677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:17.484968 22671 server_base.cc:1034] running on GCE node
I20260926 20:55:17.485692 22671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:17.487155 22671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:17.488379 22671 hybrid_clock.cc:648] HybridClock initialized: now 1790456117488341 us; error 49 us; skew 500 ppm
I20260926 20:55:17.488720 22671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:17 dist-test-slave-2dg3 krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456117, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:17.491786 22671 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:55:17.493309 22671 webserver.cc:460] Webserver started at http://127.8.249.190:35041/ using document root <none> and password file <none>
I20260926 20:55:17.494007 22671 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:17.494145 22671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:17.494462 22671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:17.497437 22671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "f0daa6bfdeb848108202425cae3d556c"
format_stamp: "Formatted at 2026-09-26 20:55:17 on dist-test-slave-2dg3"
server_key: "aa7e031ac6777fc3a4846d8807691a31"
server_key_iv: "d963b5c2c8568ca70686b6812ee3b0cf"
server_key_version: "encryptionkey@0"
I20260926 20:55:17.498261 22671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "f0daa6bfdeb848108202425cae3d556c"
format_stamp: "Formatted at 2026-09-26 20:55:17 on dist-test-slave-2dg3"
server_key: "aa7e031ac6777fc3a4846d8807691a31"
server_key_iv: "d963b5c2c8568ca70686b6812ee3b0cf"
server_key_version: "encryptionkey@0"
I20260926 20:55:17.503317 22671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 20:55:17.506637 22686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:17.507970 22671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:55:17.508140 22671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "f0daa6bfdeb848108202425cae3d556c"
format_stamp: "Formatted at 2026-09-26 20:55:17 on dist-test-slave-2dg3"
server_key: "aa7e031ac6777fc3a4846d8807691a31"
server_key_iv: "d963b5c2c8568ca70686b6812ee3b0cf"
server_key_version: "encryptionkey@0"
I20260926 20:55:17.508334 22671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:17.531253 22671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:17.535334 22671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:17.535625 22671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:17.554668 22671 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:43033
I20260926 20:55:17.554692 22737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:43033 every 8 connection(s)
I20260926 20:55:17.556129 22671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:17.556708 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 22671
I20260926 20:55:17.556936 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:55:17.557714 9190 external_mini_cluster.cc:1428] Setting key 80542930ec5d55e98eae47a22d43301b
I20260926 20:55:17.560209 22738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:17.566712 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.249.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.189@KRBTEST.COM" created.
Sep 26 20:55:17 dist-test-slave-2dg3 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456117, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:17.591871 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:17.569997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.8.249.190:43033)
0926 20:55:17.572006 (+ 2009us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:17.572012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:17.572055 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:55:17.574364 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:17.574370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:17.574602 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:17.574769 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:17.576455 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:17.577626 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:17.579237 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:17.579875 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:17.580099 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:17.583538 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:17.583551 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:17.583571 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:17.583614 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:17.587521 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:17.588455 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:17.588461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:17.588463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:17.588603 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:17.589047 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:17.589050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:17.589052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:17.589553 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:17.589708 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:17.590371 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 20:55:17.590677 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1814,"spinlock_wait_cycles":69504,"thread_start_us":1578,"threads_started":1}
I20260926 20:55:17.591907 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:17.569997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:43033 (local address 127.0.0.1:47588)
0926 20:55:17.570980 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:17.571042 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:17.573960 (+ 2918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:17.574880 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:17.574890 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:17.575743 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:17.576249 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:17.576278 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:17.577819 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:17.577828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:17.579114 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:17.579126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:17.580141 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:17.580151 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:17.580387 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:17.581094 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:17.581116 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:17.583275 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:17.587671 (+ 4396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:17.587686 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:17.587702 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:17.588317 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:17.588716 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:17.588724 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:17.588730 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:17.588939 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:17.589697 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:17.589708 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:17.589969 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 20:55:17.590252 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 20:55:17.590954 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":85,"thread_start_us":411,"threads_started":1}
I20260926 20:55:17.595008 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:17.596276 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:17.595757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46371 (local address 127.0.0.1:60570)
0926 20:55:17.595970 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:17.596111 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 20:55:17.596583 22690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
I20260926 20:55:17.598348 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:17.597857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46371 (local address 127.0.0.1:60584)
0926 20:55:17.598050 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:17.598150 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 20:55:17.598619 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:46371: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.189.keytab.
Entry for principal kudu/127.8.249.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.189.keytab.
I20260926 20:55:17.610721 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.189:46371
--webserver_interface=127.8.249.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.189.keytab
--principal=kudu/127.8.249.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:43033,127.8.249.189:46371,127.8.249.188:34391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:55:17.647930 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } attempt: 1
I20260926 20:55:17.649115 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:17.648667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46371 (local address 127.0.0.1:60598)
0926 20:55:17.648855 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:17.648991 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 20:55:17.650188 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:17.649753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46371 (local address 127.0.0.1:60608)
0926 20:55:17.649930 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:17.650032 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 20:55:17.650444 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:46371: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
I20260926 20:55:17.718662 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } attempt: 2
I20260926 20:55:17.719854 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:17.719431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46371 (local address 127.0.0.1:60614)
0926 20:55:17.719623 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:17.719713 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 20:55:17.720852 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:17.720488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46371 (local address 127.0.0.1:60630)
0926 20:55:17.720638 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:17.720724 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 20:55:17.721027 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:46371: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
I20260926 20:55:17.812312 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } attempt: 3
I20260926 20:55:17.813563 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:17.813093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46371 (local address 127.0.0.1:60644)
0926 20:55:17.813317 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:17.813430 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 20:55:17.814751 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:17.814371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46371 (local address 127.0.0.1:60646)
0926 20:55:17.814538 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:17.814609 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 20:55:17.815014 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:46371: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
W20260926 20:55:17.939641 22745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:17.939927 22745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:17.940114 22745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:17.947263 22745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:17.947407 22745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:17.947495 22745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:17.947584 22745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:17.947645 22745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:17.957729 22745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:43033,127.8.249.189:46371,127.8.249.188:34391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.189.keytab
--principal=kudu/127.8.249.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.189:46371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.249.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:17.958294 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } attempt: 4
I20260926 20:55:17.959364 22745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:17.959510 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:17.958967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46371 (local address 127.0.0.1:60648)
0926 20:55:17.959272 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:17.959356 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 20:55:17.960453 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:17.960113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46371 (local address 127.0.0.1:60662)
0926 20:55:17.960260 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:17.960350 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 20:55:17.960680 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:46371: Network error: Client connection negotiation failed: client connection to 127.8.249.189:46371: connect: Connection refused (error 111)
I20260926 20:55:17.960791 22745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:17.972486 22750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:17.972437 22751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:17.972517 22753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:17.973209 22745 server_base.cc:1034] running on GCE node
I20260926 20:55:17.973980 22745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:17.975462 22745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:17.976687 22745 hybrid_clock.cc:648] HybridClock initialized: now 1790456117976646 us; error 61 us; skew 500 ppm
I20260926 20:55:17.977039 22745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:17 dist-test-slave-2dg3 krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456117, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:17.985198 22745 init.cc:377] Logged in from keytab as kudu/127.8.249.189@KRBTEST.COM (short username kudu)
I20260926 20:55:17.986709 22745 webserver.cc:460] Webserver started at http://127.8.249.189:35829/ using document root <none> and password file <none>
I20260926 20:55:17.987447 22745 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:17.987591 22745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:17.987897 22745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:17.991319 22745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/data/instance:
uuid: "afe712d9741a4446b520b53c1a5a98d5"
format_stamp: "Formatted at 2026-09-26 20:55:17 on dist-test-slave-2dg3"
server_key: "a4a018b92ca37b96a3d03a840b6e6b75"
server_key_iv: "7bdf78174ab12c7d3b37d1126bb2418f"
server_key_version: "encryptionkey@0"
I20260926 20:55:17.992179 22745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/wal/instance:
uuid: "afe712d9741a4446b520b53c1a5a98d5"
format_stamp: "Formatted at 2026-09-26 20:55:17 on dist-test-slave-2dg3"
server_key: "a4a018b92ca37b96a3d03a840b6e6b75"
server_key_iv: "7bdf78174ab12c7d3b37d1126bb2418f"
server_key_version: "encryptionkey@0"
I20260926 20:55:17.997574 22745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:55:18.001196 22760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:18.002640 22745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:55:18.002847 22745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/wal
uuid: "afe712d9741a4446b520b53c1a5a98d5"
format_stamp: "Formatted at 2026-09-26 20:55:17 on dist-test-slave-2dg3"
server_key: "a4a018b92ca37b96a3d03a840b6e6b75"
server_key_iv: "7bdf78174ab12c7d3b37d1126bb2418f"
server_key_version: "encryptionkey@0"
I20260926 20:55:18.003052 22745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:18.015136 22745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:18.018857 22745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:18.019088 22745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:18.036635 22745 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.189:46371
I20260926 20:55:18.036657 22811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.189:46371 every 8 connection(s)
I20260926 20:55:18.038012 22745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/data/info.pb
I20260926 20:55:18.041613 22812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:18.042244 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 22745
I20260926 20:55:18.042443 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-1/wal/instance
I20260926 20:55:18.042950 9190 external_mini_cluster.cc:1428] Setting key 8e8a3293068951bc89fa10ae2144415f
I20260926 20:55:18.048663 22812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:55:18 dist-test-slave-2dg3 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456117, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.189@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.249.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.188@KRBTEST.COM" created.
I20260926 20:55:18.069633 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.051418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.8.249.190:43033)
0926 20:55:18.051658 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.051666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.052884 (+ 1218us) server_negotiation.cc:407] Connection header received
0926 20:55:18.054717 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.054726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.054857 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.054979 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.056408 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.057558 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.059159 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.059943 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.060114 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.063634 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.063639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.063642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.063674 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.066447 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.067415 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.067419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.067421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.067534 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.067991 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.067994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.067996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.068254 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.068361 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.069255 (+ 894us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.069414 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:55:18.071220 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.051888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:43033 (local address 127.0.0.1:47596)
0926 20:55:18.052702 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.052746 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:18.054448 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:18.055158 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:18.055166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:18.055685 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:18.056230 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.056247 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.057716 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.057720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.059011 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.059023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.060150 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.060154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.060350 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.061063 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:18.061084 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:18.063438 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:18.066581 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.066593 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.066622 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.067307 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.067641 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.067649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.067655 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.067885 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.068391 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:18.068401 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:18.068861 (+ 460us) client_negotiation.cc:769] Sending connection context
0926 20:55:18.069159 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 20:55:18.069697 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":378,"threads_started":1}
I20260926 20:55:18.072872 22812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:55:18 dist-test-slave-2dg3 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456117, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.189@KRBTEST.COM for kudu/127.8.249.189@KRBTEST.COM
I20260926 20:55:18.088835 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.073736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46371 (local address 127.0.0.1:60670)
0926 20:55:18.074019 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.074041 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:18.074149 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:18.075112 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:18.075123 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:18.075619 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:18.075834 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.075848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.077154 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.077162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.078712 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.078738 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.079702 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.079712 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.079966 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.080554 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:18.080580 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:18.083104 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:18.086308 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.086315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.086319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.086669 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.087077 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.087084 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.087089 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.087224 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.087810 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:18.087815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:18.087952 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 20:55:18.088106 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 20:55:18.088394 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 20:55:18.089139 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.073892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.8.249.189:46371)
0926 20:55:18.074630 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.074636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.074660 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:55:18.074701 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.074707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.074942 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.075072 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.075982 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.077017 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.078931 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.079523 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.079616 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.083319 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.083330 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.083344 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.083371 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.086170 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.086829 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.086838 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.086841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.086970 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.087343 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.087349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.087357 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.087686 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.087908 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.088231 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.088512 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":595,"thread_start_us":449,"threads_started":1}
I20260926 20:55:18.091598 22812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:18.092705 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.092235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48496)
0926 20:55:18.092453 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.092560 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 20:55:18.092944 22764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
I20260926 20:55:18.094446 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.094132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48510)
0926 20:55:18.094280 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.094348 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 20:55:18.094653 22812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:34391: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.188.keytab.
Entry for principal kudu/127.8.249.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.188.keytab.
I20260926 20:55:18.097738 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.188:34391
--webserver_interface=127.8.249.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.188.keytab
--principal=kudu/127.8.249.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:45615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:43033,127.8.249.189:46371,127.8.249.188:34391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:55:18.144104 22812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } attempt: 1
I20260926 20:55:18.145300 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.144840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48518)
0926 20:55:18.145038 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.145148 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 20:55:18.146375 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.146038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48520)
0926 20:55:18.146205 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.146274 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 20:55:18.146577 22812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:34391: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
I20260926 20:55:18.214819 22812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } attempt: 2
I20260926 20:55:18.215929 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.215496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48530)
0926 20:55:18.215689 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.215783 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 20:55:18.216872 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.216594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48532)
0926 20:55:18.216712 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.216773 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 20:55:18.217098 22812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:34391: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
I20260926 20:55:18.260002 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } attempt: 5
Sep 26 20:55:18 dist-test-slave-2dg3 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456117, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for kudu/127.8.249.189@KRBTEST.COM
I20260926 20:55:18.274863 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.261064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.8.249.189:46371)
0926 20:55:18.261255 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.261260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.261279 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:55:18.261348 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.261351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.261411 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.261497 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.262466 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.263180 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.264894 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.265321 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.265409 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.269218 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.269224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.269227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.269261 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.271944 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.272900 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.272903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.272906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.273024 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.273492 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.273495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.273497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.273770 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.273881 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.274409 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.274581 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:55:18.275254 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.260814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46371 (local address 127.0.0.1:60672)
0926 20:55:18.261055 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.261080 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:18.261220 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:18.261566 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:18.261571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:18.262065 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:18.262313 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.262326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.263310 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.263315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.264724 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.264748 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.265464 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.265470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.265666 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.266311 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:18.266335 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:18.268990 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:18.272106 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.272111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.272116 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.272761 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.273147 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.273155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.273160 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.273376 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.273901 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:18.273908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:18.274124 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 20:55:18.274329 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 20:55:18.274757 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 20:55:18.276516 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:18.277676 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.277197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48542)
0926 20:55:18.277400 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.277505 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 20:55:18.278635 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.278266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48558)
0926 20:55:18.278419 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.278515 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 20:55:18.278880 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:34391: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
I20260926 20:55:18.308302 22812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } attempt: 3
I20260926 20:55:18.309511 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.309039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48566)
0926 20:55:18.309258 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.309357 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 20:55:18.310757 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.310334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48578)
0926 20:55:18.310572 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.310645 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260926 20:55:18.311094 22812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:34391: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
I20260926 20:55:18.330123 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } attempt: 1
I20260926 20:55:18.331387 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.330856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48594)
0926 20:55:18.331073 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.331189 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 20:55:18.332480 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.332031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48610)
0926 20:55:18.332237 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.332330 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 20:55:18.332738 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:34391: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
I20260926 20:55:18.401010 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } attempt: 2
I20260926 20:55:18.402315 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.401729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48620)
0926 20:55:18.401992 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.402107 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 20:55:18.403466 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.403065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48634)
0926 20:55:18.403241 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.403329 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 20:55:18.403733 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:34391: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
W20260926 20:55:18.437065 22819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:18.437440 22819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:18.437599 22819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:18.445262 22819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:18.445441 22819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:18.445559 22819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:18.445668 22819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:18.445747 22819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:18.454319 22812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } attempt: 4
I20260926 20:55:18.455569 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.455050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48648)
0926 20:55:18.455297 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.455398 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 20:55:18.456514 22819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:45615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:43033,127.8.249.189:46371,127.8.249.188:34391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.188.keytab
--principal=kudu/127.8.249.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.188:34391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.249.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:18.456643 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.456296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48656)
0926 20:55:18.456447 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.456535 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 20:55:18.456893 22812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:34391: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
I20260926 20:55:18.457937 22819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:18.459385 22819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:18.471339 22824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:18.471376 22827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:18.471199 22825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:18.472213 22819 server_base.cc:1034] running on GCE node
I20260926 20:55:18.472990 22819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:18.474303 22819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:18.475538 22819 hybrid_clock.cc:648] HybridClock initialized: now 1790456118475499 us; error 49 us; skew 500 ppm
I20260926 20:55:18.475895 22819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:18 dist-test-slave-2dg3 krb5kdc[22658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456118, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:18.479028 22819 init.cc:377] Logged in from keytab as kudu/127.8.249.188@KRBTEST.COM (short username kudu)
I20260926 20:55:18.480641 22819 webserver.cc:460] Webserver started at http://127.8.249.188:38653/ using document root <none> and password file <none>
I20260926 20:55:18.481367 22819 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:18.481513 22819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:18.481842 22819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:18.484970 22819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/data/instance:
uuid: "76f9c4a61f9346aeb105bab2fedc381d"
format_stamp: "Formatted at 2026-09-26 20:55:18 on dist-test-slave-2dg3"
server_key: "cc7a11ac4fec33938a9e91e18078bc8f"
server_key_iv: "40d1c0d8d5b81885a7a8605bb43f6c4e"
server_key_version: "encryptionkey@0"
I20260926 20:55:18.485796 22819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/wal/instance:
uuid: "76f9c4a61f9346aeb105bab2fedc381d"
format_stamp: "Formatted at 2026-09-26 20:55:18 on dist-test-slave-2dg3"
server_key: "cc7a11ac4fec33938a9e91e18078bc8f"
server_key_iv: "40d1c0d8d5b81885a7a8605bb43f6c4e"
server_key_version: "encryptionkey@0"
I20260926 20:55:18.490931 22819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 20:55:18.494210 22834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:18.495582 22819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 20:55:18.495764 22819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/wal
uuid: "76f9c4a61f9346aeb105bab2fedc381d"
format_stamp: "Formatted at 2026-09-26 20:55:18 on dist-test-slave-2dg3"
server_key: "cc7a11ac4fec33938a9e91e18078bc8f"
server_key_iv: "40d1c0d8d5b81885a7a8605bb43f6c4e"
server_key_version: "encryptionkey@0"
I20260926 20:55:18.495968 22819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:18.509943 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } attempt: 3
I20260926 20:55:18.511067 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.510522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48670)
0926 20:55:18.510752 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.510880 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 20:55:18.511721 22819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:18.512187 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.511832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48684)
0926 20:55:18.512007 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.512082 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 20:55:18.512446 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:34391: Network error: Client connection negotiation failed: client connection to 127.8.249.188:34391: connect: Connection refused (error 111)
I20260926 20:55:18.515906 22819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:18.516177 22819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:18.534607 22819 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.188:34391
I20260926 20:55:18.534629 22885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.188:34391 every 8 connection(s)
I20260926 20:55:18.536020 22819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/data/info.pb
I20260926 20:55:18.539618 22886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:18.540522 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 22819
I20260926 20:55:18.540729 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790455916331309-9190-0/minicluster-data/master-2/wal/instance
I20260926 20:55:18.541347 9190 external_mini_cluster.cc:1428] Setting key e6503b8665c619b9a0b4bbcbaa5296a5
I20260926 20:55:18.548882 22886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:55:18 dist-test-slave-2dg3 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.188@KRBTEST.COM
Sep 26 20:55:18 dist-test-slave-2dg3 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456118, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.188@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:18.565644 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.543241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.8.249.188:34391)
0926 20:55:18.544017 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.544029 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.544068 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:55:18.545806 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.545828 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.546224 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.546571 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.548348 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.549610 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.551474 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.552294 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.552490 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.557195 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.557202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.557211 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.557251 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.560809 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.561590 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.561598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.561606 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.561802 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.562241 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.562247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.562250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.562985 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.563291 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.564141 (+ 850us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.564436 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":362,"threads_started":1}
I20260926 20:55:18.567739 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.551590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.8.249.190:43033)
0926 20:55:18.551774 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.551781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.552606 (+ 825us) server_negotiation.cc:407] Connection header received
0926 20:55:18.552880 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.552885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.552950 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.553052 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.554114 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.554964 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.556794 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.557465 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.557587 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.561575 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.561584 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.561588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.561630 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.564835 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.565629 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.565632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.565634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.565791 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.566203 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.566211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.566215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.566479 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.566648 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.567151 (+ 503us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.567320 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:55:18.567813 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.551632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:43033 (local address 127.0.0.1:47612)
0926 20:55:18.552474 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.552521 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:18.552726 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:18.553107 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:18.553114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:18.553689 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:18.553964 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.553974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.555114 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.555124 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.556650 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.556666 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.557654 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.557662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.557844 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.558463 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:18.558485 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:18.561357 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:18.564955 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.564963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.564969 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.565517 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.565924 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.565930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.565934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.566090 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.566610 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:18.566619 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:18.566855 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 20:55:18.567080 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 20:55:18.567407 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":454,"threads_started":1}
I20260926 20:55:18.569168 22886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:55:18 dist-test-slave-2dg3 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456118, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.188@KRBTEST.COM for kudu/127.8.249.189@KRBTEST.COM
Sep 26 20:55:18 dist-test-slave-2dg3 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:18.584280 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.570000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:46371 (local address 127.0.0.1:60676)
0926 20:55:18.570234 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.570251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:18.570358 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:18.570652 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:18.570658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:18.571144 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:18.571333 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.571352 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.572646 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.572651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.574072 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.574088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.575108 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.575114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.575341 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.575977 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:18.575999 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:18.578488 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:18.581619 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.581626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.581633 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.582061 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.582499 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.582502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.582505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.582637 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.583311 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:18.583318 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:18.583523 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 20:55:18.583685 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 20:55:18.583995 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260926 20:55:18.584316 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.569963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.8.249.189:46371)
0926 20:55:18.570130 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.570135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.570318 (+ 183us) server_negotiation.cc:407] Connection header received
0926 20:55:18.570459 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.570463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.570520 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.570596 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.571491 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.572516 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.574214 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.574942 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.575059 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.578722 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.578727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.578731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.578767 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.581483 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.582190 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.582196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.582199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.582380 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.582747 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.582752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.582758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.583159 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.583333 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.583762 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.583935 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:55:18.585567 22886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:55:18.586741 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.569052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.8.249.190:43033)
0926 20:55:18.569192 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.569196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.569370 (+ 174us) server_negotiation.cc:407] Connection header received
0926 20:55:18.569512 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.569515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.569569 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.569642 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.571311 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.572446 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.574290 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.574942 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.575059 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.580035 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.580040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.580044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.580077 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.583207 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.584116 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.584121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.584125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.584271 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.584716 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.584724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.584730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.585015 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.585308 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.586145 (+ 837us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.586405 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 20:55:18 dist-test-slave-2dg3 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456118, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.188@KRBTEST.COM for kudu/127.8.249.188@KRBTEST.COM
Sep 26 20:55:18 dist-test-slave-2dg3 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.189@KRBTEST.COM
I20260926 20:55:18.599350 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.586430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48712 (local address 127.8.249.188:34391)
0926 20:55:18.586569 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.586573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.586589 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:55:18.586674 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.586678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.586733 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.586866 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.587685 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.588628 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.589712 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.590128 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.590270 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.593264 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.593270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.593275 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.593314 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.596481 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.597177 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.597182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.597188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.597342 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.597689 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.597695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.597699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.598039 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.598143 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.598616 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.598985 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":183,"server-negotiator.queue_time_us":49}
I20260926 20:55:18.599375 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.586289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48712)
0926 20:55:18.586448 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.586467 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:18.586563 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:18.586895 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:18.586902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:18.587416 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:18.587545 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.587553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.588754 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.588761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.589582 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.589592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.590281 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.590286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.590532 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.591156 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:18.591172 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:18.593088 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:18.596626 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.596631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.596635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.597072 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.597456 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.597459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.597461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.597581 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.598155 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:18.598161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:18.598325 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 20:55:18.598499 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 20:55:18.598841 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":23}
I20260926 20:55:18.604216 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.588512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.8.249.189:46371)
0926 20:55:18.588670 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.588676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.588693 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:55:18.588865 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.588870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.588939 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.589034 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.590260 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.591030 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.592229 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.592679 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.592816 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.597941 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.597945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.597949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.597978 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.601589 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.602260 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.602267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.602270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.602408 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.602772 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.602778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.602781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.603059 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.603334 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.603657 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.603896 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:55:18.604363 22886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d: Bootstrap starting.
I20260926 20:55:18.608516 22886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:18.609200 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:55:18.609812 22886 log.cc:824] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:18.613096 22886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d: No bootstrap required, opened a new log
I20260926 20:55:18.624804 22886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } }
I20260926 20:55:18.625293 22886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:18.625427 22886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f9c4a61f9346aeb105bab2fedc381d, State: Initialized, Role: FOLLOWER
I20260926 20:55:18.626319 22886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } }
I20260926 20:55:18.633450 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } } }
I20260926 20:55:18.633884 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:18.635980 22886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:18.637779 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.617026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.8.249.189:46371)
0926 20:55:18.617193 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.617199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.617792 (+ 593us) server_negotiation.cc:407] Connection header received
0926 20:55:18.618060 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.618066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.618149 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.618232 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.620641 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.621763 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.624023 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.624639 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.624776 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.629637 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.629648 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.629655 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.629719 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.634065 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.635052 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.635057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.635061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.635237 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.635736 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.635743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.635746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.636140 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.636382 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.636988 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.637267 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:55:18.642127 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.616976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.8.249.190:43033)
0926 20:55:18.617180 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.617190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.622180 (+ 4990us) server_negotiation.cc:407] Connection header received
0926 20:55:18.622691 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.622700 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.622824 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.622914 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.626224 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.627378 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.629663 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.630315 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.630409 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.634397 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.634405 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.634412 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.634473 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.638378 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.639423 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.639430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.639436 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.639598 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.640038 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.640043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.640047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.640426 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.640642 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.641314 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.641633 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:55:18.643864 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.617881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48722 (local address 127.8.249.188:34391)
0926 20:55:18.618132 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.618141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.626031 (+ 7890us) server_negotiation.cc:407] Connection header received
0926 20:55:18.626432 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.626437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.626520 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.626589 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.628777 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.629860 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.631904 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.632650 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.632782 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.636984 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.636993 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.636999 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.637039 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.640623 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.641642 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.641646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.641648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.641774 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.642172 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.642181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.642183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.642547 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.642815 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.643265 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.643487 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 20:55:18.653285 22910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:55:18.653430 22910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:55:18.649060 22819 master_runner.cc:186] Error getting master registration for 127.8.249.190:43033: OK, instance_id { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" instance_seqno: 1790456117549129 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:55:18.657330 22819 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:55:18.689653 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } attempt: 4
Sep 26 20:55:18 dist-test-slave-2dg3 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456117, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for kudu/127.8.249.188@KRBTEST.COM
I20260926 20:55:18.704761 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.690313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48726 (local address 127.8.249.188:34391)
0926 20:55:18.690508 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.690514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.690534 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:55:18.690712 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.690719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.690828 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.690930 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.691721 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.692913 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.694171 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.694884 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.695021 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.698625 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.698631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.698637 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.698676 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.701873 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.702635 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.702645 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.702649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.702856 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.703260 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.703266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.703270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.703570 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.703725 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.704142 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.704380 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:55:18.704779 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.690229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48726)
0926 20:55:18.690454 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.690474 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:18.690575 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:18.690961 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:18.690965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:18.691383 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:18.691555 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.691564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.693071 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.693079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.694000 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.694012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.695037 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.695042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.695184 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.695830 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:18.695846 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:18.698392 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:18.702041 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.702048 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.702053 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.702497 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.702990 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.702995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.702999 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.703145 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.703701 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:18.703706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:18.703877 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 20:55:18.704025 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 20:55:18.704336 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 20:55:18.710002 22738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c: Bootstrap starting.
I20260926 20:55:18.714262 22738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:18.715585 22738 log.cc:824] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:18.718430 22738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c: No bootstrap required, opened a new log
I20260926 20:55:18.725903 22738 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } }
I20260926 20:55:18.726339 22738 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:18.726468 22738 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0daa6bfdeb848108202425cae3d556c, State: Initialized, Role: FOLLOWER
I20260926 20:55:18.727241 22738 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } }
I20260926 20:55:18.728332 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } } }
I20260926 20:55:18.728711 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:18.729663 22738 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:55:18.739655 22924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:55:18.739789 22924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:55:18.738718 22671 master_runner.cc:186] Error getting master registration for 127.8.249.189:46371: OK, instance_id { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" instance_seqno: 1790456118031463 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:55:18.747663 22671 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:55:18.756116 22812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } attempt: 5
Sep 26 20:55:18 dist-test-slave-2dg3 krb5kdc[22658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456117, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.189@KRBTEST.COM for kudu/127.8.249.188@KRBTEST.COM
W20260926 20:55:18.768227 22840 tablet.cc:2369] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:18.770867 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.756755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:34391 (local address 127.0.0.1:48732)
0926 20:55:18.757056 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:18.757077 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:18.757206 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:18.757600 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:18.757605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:18.758047 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:18.758203 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.758212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.759323 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.759329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.760587 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:18.760600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.761676 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.761684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:18.761891 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.762972 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:18.762993 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:18.765103 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:18.768422 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.768427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.768431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.768815 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.769184 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:18.769190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:18.769193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:18.769336 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:18.769856 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:18.769862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:18.770008 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 20:55:18.770159 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 20:55:18.770439 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260926 20:55:18.770890 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:18.756911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48732 (local address 127.8.249.188:34391)
0926 20:55:18.757080 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:18.757085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:18.757130 (+ 45us) server_negotiation.cc:407] Connection header received
0926 20:55:18.757374 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:18.757382 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:18.757443 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:18.757577 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:18.758379 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.759186 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.760741 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:18.761372 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:18.761492 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:18.765275 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:18.765280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:18.765283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:18.765314 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:18.768228 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.768938 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.768943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.768947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.769076 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:18.769438 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:18.769443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:18.769446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:18.769727 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:18.769892 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:18.770252 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 20:55:18.770466 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:55:18.775527 22812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5: Bootstrap starting.
I20260926 20:55:18.780435 22812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:18.781543 22812 log.cc:824] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:18.784286 22812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5: No bootstrap required, opened a new log
W20260926 20:55:18.788456 22692 tablet.cc:2369] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:18.791916 22812 raft_consensus.cc:348] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } }
I20260926 20:55:18.792299 22812 raft_consensus.cc:374] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:18.792441 22812 raft_consensus.cc:729] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe712d9741a4446b520b53c1a5a98d5, State: Initialized, Role: FOLLOWER
I20260926 20:55:18.793076 22812 consensus_queue.cc:260] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } }
I20260926 20:55:18.798751 22927 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } } }
I20260926 20:55:18.799206 22927 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:18.800246 22812 sys_catalog.cc:564] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:18.817415 22745 master_runner.cc:263] No leader master found from master afe712d9741a4446b520b53c1a5a98d5
I20260926 20:55:18.817588 22745 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 20:55:18.821434 22938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:55:18.821600 22938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:55:18.895360 22913 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:55:18.895712 22913 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } }
I20260926 20:55:18.897405 22913 leader_election.cc:288] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afe712d9741a4446b520b53c1a5a98d5 (127.8.249.189:46371), 76f9c4a61f9346aeb105bab2fedc381d (127.8.249.188:34391)
I20260926 20:55:18.898659 22787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0daa6bfdeb848108202425cae3d556c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe712d9741a4446b520b53c1a5a98d5" is_pre_election: true
I20260926 20:55:18.899359 22787 raft_consensus.cc:2454] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0daa6bfdeb848108202425cae3d556c in term 0.
I20260926 20:55:18.900313 22690 leader_election.cc:302] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afe712d9741a4446b520b53c1a5a98d5, f0daa6bfdeb848108202425cae3d556c; no voters:
I20260926 20:55:18.901542 22861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0daa6bfdeb848108202425cae3d556c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f9c4a61f9346aeb105bab2fedc381d" is_pre_election: true
I20260926 20:55:18.902134 22861 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0daa6bfdeb848108202425cae3d556c in term 0.
I20260926 20:55:18.903160 22913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:55:18.903307 22913 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:55:18.903404 22913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:18.911372 22913 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } }
I20260926 20:55:18.915977 22787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0daa6bfdeb848108202425cae3d556c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe712d9741a4446b520b53c1a5a98d5"
I20260926 20:55:18.916206 22787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:18.919777 22787 raft_consensus.cc:2454] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0daa6bfdeb848108202425cae3d556c in term 1.
I20260926 20:55:18.920284 22690 leader_election.cc:302] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afe712d9741a4446b520b53c1a5a98d5, f0daa6bfdeb848108202425cae3d556c; no voters:
I20260926 20:55:18.923110 22913 leader_election.cc:288] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [CANDIDATE]: Term 1 election: Requested vote from peers afe712d9741a4446b520b53c1a5a98d5 (127.8.249.189:46371), 76f9c4a61f9346aeb105bab2fedc381d (127.8.249.188:34391)
I20260926 20:55:18.923667 22913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:18.924085 22913 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [term 1 LEADER]: Becoming Leader. State: Replica: f0daa6bfdeb848108202425cae3d556c, State: Running, Role: LEADER
I20260926 20:55:18.924889 22913 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } }
I20260926 20:55:18.939448 22861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0daa6bfdeb848108202425cae3d556c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f9c4a61f9346aeb105bab2fedc381d"
I20260926 20:55:18.939724 22861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:18.943243 22861 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0daa6bfdeb848108202425cae3d556c in term 1.
I20260926 20:55:18.948786 22940 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0daa6bfdeb848108202425cae3d556c. Latest consensus state: current_term: 1 leader_uuid: "f0daa6bfdeb848108202425cae3d556c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } } }
I20260926 20:55:18.949350 22940 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:18.967464 22942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:18.969748 22942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:18.985698 22861 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [term 1 FOLLOWER]: Refusing update from remote peer f0daa6bfdeb848108202425cae3d556c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:18.986953 22940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [LEADER]: Connected to new peer: Peer: permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:18.994858 22787 raft_consensus.cc:1261] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [term 1 FOLLOWER]: Refusing update from remote peer f0daa6bfdeb848108202425cae3d556c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:18.995977 22940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [LEADER]: Connected to new peer: Peer: permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:19.014326 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0daa6bfdeb848108202425cae3d556c. Latest consensus state: current_term: 1 leader_uuid: "f0daa6bfdeb848108202425cae3d556c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } } }
I20260926 20:55:19.014654 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:55:19.021517 22766 tablet.cc:2369] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:19.022993 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f0daa6bfdeb848108202425cae3d556c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } } }
I20260926 20:55:19.023247 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:19.024107 22943 mvcc.cc:205] Tried to move back new op lower bound from 7333708263351652352 to 7333708263127556096. Current Snapshot: MvccSnapshot[applied={T|T < 7333708263351652352}]
I20260926 20:55:19.025615 22927 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0daa6bfdeb848108202425cae3d556c. Latest consensus state: current_term: 1 leader_uuid: "f0daa6bfdeb848108202425cae3d556c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } } }
I20260926 20:55:19.025924 22927 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:19.029065 22927 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f0daa6bfdeb848108202425cae3d556c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } } }
I20260926 20:55:19.029410 22927 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:55:19.035562 22940 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f0daa6bfdeb848108202425cae3d556c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } } }
I20260926 20:55:19.035904 22940 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:19.037539 22942 catalog_manager.cc:1347] Generated new cluster ID: 70eec412cc7d47669c6092d3f2933472
I20260926 20:55:19.037626 22942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:55:19.045918 22940 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f0daa6bfdeb848108202425cae3d556c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0daa6bfdeb848108202425cae3d556c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 43033 } } peers { permanent_uuid: "afe712d9741a4446b520b53c1a5a98d5" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 46371 } } peers { permanent_uuid: "76f9c4a61f9346aeb105bab2fedc381d" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 34391 } } }
I20260926 20:55:19.046250 22940 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:19.066643 22942 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:55:19.068579 22942 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:19.088402 22942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0daa6bfdeb848108202425cae3d556c: Generated new TSK 0
I20260926 20:55:19.089677 22942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:55:19.655404 22910 catalog_manager.cc:1259] Loaded cluster ID: 70eec412cc7d47669c6092d3f2933472
I20260926 20:55:19.655503 22910 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d: loading cluster ID for follower catalog manager: success
I20260926 20:55:19.657840 22910 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d: acquiring CA information for follower catalog manager: success
I20260926 20:55:19.659590 22910 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 76f9c4a61f9346aeb105bab2fedc381d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:55:19.824697 22938 catalog_manager.cc:1259] Loaded cluster ID: 70eec412cc7d47669c6092d3f2933472
I20260926 20:55:19.824833 22938 catalog_manager.cc:1552] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5: loading cluster ID for follower catalog manager: success
I20260926 20:55:19.828476 22938 catalog_manager.cc:1574] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5: acquiring CA information for follower catalog manager: success
I20260926 20:55:19.831065 22938 catalog_manager.cc:1602] T 00000000000000000000000000000000 P afe712d9741a4446b520b53c1a5a98d5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:55:20.229368 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 22671
I20260926 20:55:20.230640 22733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:20.340883 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 22671
I20260926 20:55:20.361901 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 22745
I20260926 20:55:20.363181 22807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:20.469401 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 22745
I20260926 20:55:20.489727 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 22819
I20260926 20:55:20.490764 22881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:20.596161 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 22819
2026-09-26T20:55:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4820 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:55:20 dist-test-slave-2dg3 krb5kdc[22961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:55:20 dist-test-slave-2dg3 krb5kdc[22961](info): set up 2 sockets
Sep 26 20:55:20 dist-test-slave-2dg3 krb5kdc[22961](info): commencing operation
krb5kdc: starting...
W20260926 20:55:21.691395 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:55:21 dist-test-slave-2dg3 krb5kdc[22961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:55:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:55:21.928349 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36613
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:36613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:22.265530 22974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:22.265831 22974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:22.266011 22974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:22.273367 22974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:22.273483 22974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:22.273571 22974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:22.273677 22974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:22.273751 22974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:22.283754 22974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:36613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:22.285187 22974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:22.286589 22974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:22.298698 22979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:22.298786 22982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:22.299015 22980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:22.300014 22974 server_base.cc:1034] running on GCE node
I20260926 20:55:22.300880 22974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:22.302393 22974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:22.303628 22974 hybrid_clock.cc:648] HybridClock initialized: now 1790456122303574 us; error 43 us; skew 500 ppm
I20260926 20:55:22.304008 22974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:22 dist-test-slave-2dg3 krb5kdc[22961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456122, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:22.307320 22974 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:55:22.308904 22974 webserver.cc:460] Webserver started at http://127.8.249.190:35747/ using document root <none> and password file <none>
I20260926 20:55:22.309712 22974 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:22.309860 22974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:22.310191 22974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:22.313293 22974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "9d27d1dd278a4cb7a4c7c0285e5ebdfc"
format_stamp: "Formatted at 2026-09-26 20:55:22 on dist-test-slave-2dg3"
server_key: "7a4d0e8cccdf63aef58937fc6c3c56dc"
server_key_iv: "ecd56a6217e50ec987207b21398fafcc"
server_key_version: "encryptionkey@0"
I20260926 20:55:22.314095 22974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "9d27d1dd278a4cb7a4c7c0285e5ebdfc"
format_stamp: "Formatted at 2026-09-26 20:55:22 on dist-test-slave-2dg3"
server_key: "7a4d0e8cccdf63aef58937fc6c3c56dc"
server_key_iv: "ecd56a6217e50ec987207b21398fafcc"
server_key_version: "encryptionkey@0"
I20260926 20:55:22.319516 22974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 20:55:22.323033 22989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:22.324358 22974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:55:22.324543 22974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "9d27d1dd278a4cb7a4c7c0285e5ebdfc"
format_stamp: "Formatted at 2026-09-26 20:55:22 on dist-test-slave-2dg3"
server_key: "7a4d0e8cccdf63aef58937fc6c3c56dc"
server_key_iv: "ecd56a6217e50ec987207b21398fafcc"
server_key_version: "encryptionkey@0"
I20260926 20:55:22.324767 22974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:22.354497 22974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:22.359275 22974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:22.359586 22974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:22.378154 22974 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:36613
I20260926 20:55:22.378188 23040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:36613 every 8 connection(s)
I20260926 20:55:22.379544 22974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:22.381150 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 22974
I20260926 20:55:22.381366 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:55:22.382184 9190 external_mini_cluster.cc:1428] Setting key 506724a6e6f54984dfa31dd646167cf6
I20260926 20:55:22.384225 23041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:22.395081 23041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc: Bootstrap starting.
Sep 26 20:55:22 dist-test-slave-2dg3 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:22.398227 23041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:22.399462 23041 log.cc:824] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:22.402360 23041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc: No bootstrap required, opened a new log
I20260926 20:55:22.404719 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:22.384183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.8.249.190:36613)
0926 20:55:22.384949 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:22.384961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:22.385002 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:55:22.386616 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:22.386636 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:22.387109 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:22.387509 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:22.389028 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:22.390317 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:22.391686 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:22.392384 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:22.392619 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:22.396818 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:22.396826 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:22.396840 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:22.396886 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:22.400149 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:22.400960 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:22.400967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:22.400974 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:22.401144 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:22.401538 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:22.401543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:22.401548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:22.402054 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:22.402402 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:22.402864 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 20:55:22.403349 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":341,"threads_started":1}
I20260926 20:55:22.408682 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:55:22.411438 23041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d27d1dd278a4cb7a4c7c0285e5ebdfc" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36613 } }
I20260926 20:55:22.411880 23041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:22.412012 23041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d27d1dd278a4cb7a4c7c0285e5ebdfc, State: Initialized, Role: FOLLOWER
I20260926 20:55:22.412632 23041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d27d1dd278a4cb7a4c7c0285e5ebdfc" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36613 } }
I20260926 20:55:22.412854 23041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:55:22.412981 23041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:55:22.413158 23041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:22.415421 23041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d27d1dd278a4cb7a4c7c0285e5ebdfc" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36613 } }
I20260926 20:55:22.415938 23041 leader_election.cc:302] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d27d1dd278a4cb7a4c7c0285e5ebdfc; no voters:
I20260926 20:55:22.416538 23041 leader_election.cc:288] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:55:22.416671 23046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:22.417385 23046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [term 1 LEADER]: Becoming Leader. State: Replica: 9d27d1dd278a4cb7a4c7c0285e5ebdfc, State: Running, Role: LEADER
I20260926 20:55:22.417881 23046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d27d1dd278a4cb7a4c7c0285e5ebdfc" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36613 } }
I20260926 20:55:22.418610 23041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:22.420719 23048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d27d1dd278a4cb7a4c7c0285e5ebdfc. Latest consensus state: current_term: 1 leader_uuid: "9d27d1dd278a4cb7a4c7c0285e5ebdfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d27d1dd278a4cb7a4c7c0285e5ebdfc" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36613 } } }
I20260926 20:55:22.420709 23047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d27d1dd278a4cb7a4c7c0285e5ebdfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d27d1dd278a4cb7a4c7c0285e5ebdfc" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36613 } } }
I20260926 20:55:22.421248 23047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:22.421248 23048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:22.423627 23052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:22.429409 23052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:22.439728 23052 catalog_manager.cc:1347] Generated new cluster ID: 1e444597d10b403498886c500cbbb092
I20260926 20:55:22.439849 23052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:55:22.475792 23052 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:55:22.477231 23052 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:22.489042 23052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d27d1dd278a4cb7a4c7c0285e5ebdfc: Generated new TSK 0
I20260926 20:55:22.490185 23052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:55:22.520675 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 22974
I20260926 20:55:22.521904 23036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:22.617837 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 22974
2026-09-26T20:55:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2018 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:55:22 dist-test-slave-2dg3 krb5kdc[23069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:55:22 dist-test-slave-2dg3 krb5kdc[23069](info): set up 2 sockets
Sep 26 20:55:22 dist-test-slave-2dg3 krb5kdc[23069](info): commencing operation
krb5kdc: starting...
W20260926 20:55:23.719394 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:55:23 dist-test-slave-2dg3 krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:55:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:55:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:55:23.954452 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:40915
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:40915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:24.281931 23082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:24.282311 23082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:24.282472 23082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:24.290087 23082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:24.290266 23082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:24.290383 23082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:24.290488 23082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:24.290566 23082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:24.300812 23082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:40915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:40915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:24.302210 23082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:24.303551 23082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:24.315055 23087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:24.315171 23088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:24.315325 23090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:24.316222 23082 server_base.cc:1034] running on GCE node
I20260926 20:55:24.316995 23082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:24.318334 23082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:24.319527 23082 hybrid_clock.cc:648] HybridClock initialized: now 1790456124319474 us; error 44 us; skew 500 ppm
I20260926 20:55:24.319869 23082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:24 dist-test-slave-2dg3 krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456124, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:24.328501 23082 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:55:24.330114 23082 webserver.cc:460] Webserver started at http://127.8.249.190:46801/ using document root <none> and password file <none>
I20260926 20:55:24.330871 23082 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:24.331014 23082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:24.331324 23082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:24.334266 23082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "c3d47f285d274861a613789d7fbcf84f"
format_stamp: "Formatted at 2026-09-26 20:55:24 on dist-test-slave-2dg3"
server_key: "5d6892167ff953a4dfd5804ac9795003"
server_key_iv: "2b651e41d6f43b6a869e5a3467279dd4"
server_key_version: "encryptionkey@0"
I20260926 20:55:24.335076 23082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "c3d47f285d274861a613789d7fbcf84f"
format_stamp: "Formatted at 2026-09-26 20:55:24 on dist-test-slave-2dg3"
server_key: "5d6892167ff953a4dfd5804ac9795003"
server_key_iv: "2b651e41d6f43b6a869e5a3467279dd4"
server_key_version: "encryptionkey@0"
I20260926 20:55:24.340193 23082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:55:24.343673 23097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:24.344982 23082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:55:24.345165 23082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "c3d47f285d274861a613789d7fbcf84f"
format_stamp: "Formatted at 2026-09-26 20:55:24 on dist-test-slave-2dg3"
server_key: "5d6892167ff953a4dfd5804ac9795003"
server_key_iv: "2b651e41d6f43b6a869e5a3467279dd4"
server_key_version: "encryptionkey@0"
I20260926 20:55:24.345356 23082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:24.367421 23082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:24.371861 23082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:24.372156 23082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:24.391140 23082 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:40915
I20260926 20:55:24.391162 23148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:40915 every 8 connection(s)
I20260926 20:55:24.392504 23082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:24.396289 23149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:24.397033 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 23082
I20260926 20:55:24.397246 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:55:24.397889 9190 external_mini_cluster.cc:1428] Setting key 7742b83c55d3798ef5ffaa60e3537a29
I20260926 20:55:24.410737 23149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Bootstrap starting.
Sep 26 20:55:24 dist-test-slave-2dg3 krb5kdc[23069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:24.414173 23149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:24.415524 23149 log.cc:824] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:24.418517 23149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: No bootstrap required, opened a new log
I20260926 20:55:24.421615 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:24.399966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.8.249.190:40915)
0926 20:55:24.400789 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:24.400809 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:24.400863 (+ 54us) server_negotiation.cc:407] Connection header received
0926 20:55:24.402962 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:24.402992 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:24.403538 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:24.404008 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:24.405302 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:24.407339 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:24.408635 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:24.409688 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:24.409864 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:24.414099 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:24.414109 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:24.414127 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:24.414182 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:24.417513 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:24.418217 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:24.418224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:24.418231 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:24.418388 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:24.418832 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:24.418838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:24.418843 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:24.419361 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:24.419696 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:24.420094 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 20:55:24.420552 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":331,"threads_started":1}
I20260926 20:55:24.426908 23149 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } }
I20260926 20:55:24.427307 23149 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:24.427464 23149 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d47f285d274861a613789d7fbcf84f, State: Initialized, Role: FOLLOWER
I20260926 20:55:24.428087 23149 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } }
I20260926 20:55:24.428330 23149 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:55:24.428490 23149 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:55:24.428671 23149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:24.431372 23149 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } }
I20260926 20:55:24.431890 23149 leader_election.cc:302] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3d47f285d274861a613789d7fbcf84f; no voters:
I20260926 20:55:24.432478 23149 leader_election.cc:288] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:55:24.432631 23154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:24.433275 23154 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 1 LEADER]: Becoming Leader. State: Replica: c3d47f285d274861a613789d7fbcf84f, State: Running, Role: LEADER
I20260926 20:55:24.433835 23154 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } }
I20260926 20:55:24.434623 23149 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:24.436563 23156 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3d47f285d274861a613789d7fbcf84f. Latest consensus state: current_term: 1 leader_uuid: "c3d47f285d274861a613789d7fbcf84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } } }
I20260926 20:55:24.436892 23156 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:24.437459 23155 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3d47f285d274861a613789d7fbcf84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } } }
I20260926 20:55:24.437726 23155 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:24.439682 23159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:24.444607 23159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:24.454051 23159 catalog_manager.cc:1347] Generated new cluster ID: 3e4c427df9f34a34bb598376c69c56a1
I20260926 20:55:24.454200 23159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:55:24.471446 23159 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:55:24.472781 23159 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:24.479650 23159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Generated new TSK 0
I20260926 20:55:24.480582 23159 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:55:24.546716 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40915
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:24.887009 23175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:24.887291 23175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:24.887497 23175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:24.894965 23175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:24.895080 23175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:24.895277 23175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:55:24.905309 23175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:24.906646 23175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:24.908016 23175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:24.920871 23180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:24.920832 23181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:24.920923 23183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:24.922363 23175 server_base.cc:1034] running on GCE node
I20260926 20:55:24.923113 23175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:24.924098 23175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:24.925307 23175 hybrid_clock.cc:648] HybridClock initialized: now 1790456124925264 us; error 29 us; skew 500 ppm
I20260926 20:55:24.925642 23175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:24 dist-test-slave-2dg3 krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456124, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:24.928776 23175 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:55:24.930243 23175 webserver.cc:460] Webserver started at http://127.8.249.129:37165/ using document root <none> and password file <none>
I20260926 20:55:24.930985 23175 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:24.931090 23175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:24.931331 23175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:24.934077 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "a4fae9e9222c48cbaf2c6e34450ee724"
format_stamp: "Formatted at 2026-09-26 20:55:24 on dist-test-slave-2dg3"
server_key: "f44c777d5957e8ded95e50b7502ea58c"
server_key_iv: "ebb80d8037e3fd3d08d6e723499bf646"
server_key_version: "encryptionkey@0"
I20260926 20:55:24.934887 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "a4fae9e9222c48cbaf2c6e34450ee724"
format_stamp: "Formatted at 2026-09-26 20:55:24 on dist-test-slave-2dg3"
server_key: "f44c777d5957e8ded95e50b7502ea58c"
server_key_iv: "ebb80d8037e3fd3d08d6e723499bf646"
server_key_version: "encryptionkey@0"
I20260926 20:55:24.939747 23175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260926 20:55:24.943038 23190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:24.944267 23175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:55:24.944437 23175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "a4fae9e9222c48cbaf2c6e34450ee724"
format_stamp: "Formatted at 2026-09-26 20:55:24 on dist-test-slave-2dg3"
server_key: "f44c777d5957e8ded95e50b7502ea58c"
server_key_iv: "ebb80d8037e3fd3d08d6e723499bf646"
server_key_version: "encryptionkey@0"
I20260926 20:55:24.944648 23175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:24.960119 23175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:24.964427 23175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:24.964723 23175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:24.965626 23175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:24.967038 23175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:24.967129 23175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:24.967274 23175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:24.967344 23175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:25.000424 23175 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:33765
I20260926 20:55:25.000469 23302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:33765 every 8 connection(s)
I20260926 20:55:25.001746 23175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:55:25.002983 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 23175
I20260926 20:55:25.003219 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:55:25.003985 9190 external_mini_cluster.cc:1428] Setting key de665d57737dc2f4f3747a9d7a048fa6
Sep 26 20:55:25 dist-test-slave-2dg3 krb5kdc[23069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456124, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:25.022035 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:25.004659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:48597 (local address 127.8.249.190:40915)
0926 20:55:25.005086 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:25.005090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:25.006256 (+ 1166us) server_negotiation.cc:407] Connection header received
0926 20:55:25.008149 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:25.008155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:25.008256 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:25.008395 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:25.009862 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:25.010550 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:25.012095 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:25.012735 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:25.012871 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:25.016467 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:25.016471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:25.016474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:25.016504 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:25.019116 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:25.019789 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:25.019795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:25.019798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:25.019916 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:25.020348 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:25.020353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:25.020356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:25.020628 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:25.020788 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:25.021456 (+ 668us) server_negotiation.cc:299] Negotiation successful
0926 20:55:25.021671 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":215,"threads_started":1}
I20260926 20:55:25.023020 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:25.005282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:48597)
0926 20:55:25.006103 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:25.006157 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:25.007863 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:25.008635 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:25.008644 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:25.009159 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:25.009705 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:25.009722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:25.010698 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:25.010709 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:25.011951 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:25.011961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:25.012953 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:25.012958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:25.013161 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:25.013852 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:25.013875 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:25.016257 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:25.019249 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:25.019257 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:25.019272 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:25.019670 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:25.020017 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:25.020021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:25.020025 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:25.020236 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:25.020763 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:25.020770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:25.021102 (+ 332us) client_negotiation.cc:769] Sending connection context
0926 20:55:25.021357 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 20:55:25.021866 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":291,"threads_started":1}
I20260926 20:55:25.024561 23303 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40915
I20260926 20:55:25.024864 23303 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:25.025552 23303 heartbeater.cc:511] Master 127.8.249.190:40915 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
I20260926 20:55:25.027705 23114 ts_manager.cc:194] Registered new tserver with Master: a4fae9e9222c48cbaf2c6e34450ee724 (127.8.249.129:33765)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
I20260926 20:55:25.029238 23114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:48597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:55:25.054171 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40915
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:25.383791 23309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:25.384068 23309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:25.384191 23309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:25.391932 23309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:25.392079 23309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:25.392264 23309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:55:25.407799 23309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:25.409438 23309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:25.411137 23309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:25.424644 23314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:25.424556 23317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:25.424569 23315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:25.425786 23309 server_base.cc:1034] running on GCE node
I20260926 20:55:25.426397 23309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:25.427449 23309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:25.428675 23309 hybrid_clock.cc:648] HybridClock initialized: now 1790456125428636 us; error 41 us; skew 500 ppm
I20260926 20:55:25.428996 23309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:25 dist-test-slave-2dg3 krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456125, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:25.432142 23309 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:55:25.433586 23309 webserver.cc:460] Webserver started at http://127.8.249.130:39071/ using document root <none> and password file <none>
I20260926 20:55:25.434299 23309 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:25.434445 23309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:25.434746 23309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:25.437868 23309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "d992231bfaab4bf7a329e37633a4a30a"
format_stamp: "Formatted at 2026-09-26 20:55:25 on dist-test-slave-2dg3"
server_key: "ed62dfab1cce587383a682462666a6b1"
server_key_iv: "099fc7c9c2d6a0129e65c0bdee472198"
server_key_version: "encryptionkey@0"
I20260926 20:55:25.438675 23309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "d992231bfaab4bf7a329e37633a4a30a"
format_stamp: "Formatted at 2026-09-26 20:55:25 on dist-test-slave-2dg3"
server_key: "ed62dfab1cce587383a682462666a6b1"
server_key_iv: "099fc7c9c2d6a0129e65c0bdee472198"
server_key_version: "encryptionkey@0"
I20260926 20:55:25.444211 23309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 20:55:25.447992 23324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:25.449563 23309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:55:25.449741 23309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "d992231bfaab4bf7a329e37633a4a30a"
format_stamp: "Formatted at 2026-09-26 20:55:25 on dist-test-slave-2dg3"
server_key: "ed62dfab1cce587383a682462666a6b1"
server_key_iv: "099fc7c9c2d6a0129e65c0bdee472198"
server_key_version: "encryptionkey@0"
I20260926 20:55:25.449944 23309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:25.483263 23309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:25.487497 23309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:25.487798 23309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:25.488788 23309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:25.490264 23309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:25.490365 23309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:25.490522 23309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:25.490603 23309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:25.523696 23309 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:35155
I20260926 20:55:25.523744 23436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:35155 every 8 connection(s)
I20260926 20:55:25.525048 23309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:55:25.527330 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 23309
I20260926 20:55:25.527539 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:55:25.528107 9190 external_mini_cluster.cc:1428] Setting key c748f58136e47259a98ca86c0c4c8c9b
Sep 26 20:55:25 dist-test-slave-2dg3 krb5kdc[23069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456125, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:25.546999 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:25.527663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:39077 (local address 127.8.249.190:40915)
0926 20:55:25.528125 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:25.528131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:25.529235 (+ 1104us) server_negotiation.cc:407] Connection header received
0926 20:55:25.531185 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:25.531194 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:25.531281 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:25.531392 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:25.532843 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:25.534027 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:25.535719 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:25.536538 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:25.536679 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:25.540790 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:25.540795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:25.540799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:25.540835 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:25.543814 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:25.544625 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:25.544635 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:25.544637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:25.544770 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:25.545207 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:25.545216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:25.545218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:25.545488 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:25.545642 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:25.546361 (+ 719us) server_negotiation.cc:299] Negotiation successful
0926 20:55:25.546588 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":210,"threads_started":1}
I20260926 20:55:25.548112 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:25.528311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:39077)
0926 20:55:25.529075 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:25.529127 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:25.530906 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:25.531603 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:25.531611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:25.532092 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:25.532677 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:25.532694 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:25.534163 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:25.534166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:25.535568 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:25.535580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:25.536738 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:25.536743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:25.536939 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:25.537672 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:25.537712 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:25.540526 (+ 2814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:25.543974 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:25.543984 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:25.544004 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:25.544474 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:25.544884 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:25.544889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:25.544892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:25.545083 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:25.545613 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:25.545621 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:25.545991 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 20:55:25.546276 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 20:55:25.546852 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":331,"threads_started":1}
I20260926 20:55:25.549886 23437 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40915
I20260926 20:55:25.550199 23437 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:25.551033 23437 heartbeater.cc:511] Master 127.8.249.190:40915 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
I20260926 20:55:25.552525 23114 ts_manager.cc:194] Registered new tserver with Master: d992231bfaab4bf7a329e37633a4a30a (127.8.249.130:35155)
I20260926 20:55:25.553467 23114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:39077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:55:25.583814 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40915
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:25.914565 23443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:25.914944 23443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:25.915122 23443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:25.922981 23443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:25.923138 23443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:25.923324 23443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:55:25.933758 23443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:40915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:25.935377 23443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:25.936736 23443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:25.950541 23448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:25.951192 23449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:25.951870 23451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:25.952394 23443 server_base.cc:1034] running on GCE node
I20260926 20:55:25.953289 23443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:25.954926 23443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:25.956168 23443 hybrid_clock.cc:648] HybridClock initialized: now 1790456125956123 us; error 36 us; skew 500 ppm
I20260926 20:55:25.956537 23443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:25 dist-test-slave-2dg3 krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456125, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:25.959831 23443 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:55:25.961294 23443 webserver.cc:460] Webserver started at http://127.8.249.131:38329/ using document root <none> and password file <none>
I20260926 20:55:25.961978 23443 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:25.962080 23443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:25.962325 23443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:25.965890 23443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "7132b5eaf5ba4112ae5063e93d52b4e3"
format_stamp: "Formatted at 2026-09-26 20:55:25 on dist-test-slave-2dg3"
server_key: "51205398d2bee1c481f88c212f0a7c15"
server_key_iv: "89545a52e0880eeb1bd05eb9274d3b9a"
server_key_version: "encryptionkey@0"
I20260926 20:55:25.966771 23443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "7132b5eaf5ba4112ae5063e93d52b4e3"
format_stamp: "Formatted at 2026-09-26 20:55:25 on dist-test-slave-2dg3"
server_key: "51205398d2bee1c481f88c212f0a7c15"
server_key_iv: "89545a52e0880eeb1bd05eb9274d3b9a"
server_key_version: "encryptionkey@0"
I20260926 20:55:25.972157 23443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:55:25.975625 23458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:25.977017 23443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:55:25.977201 23443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "7132b5eaf5ba4112ae5063e93d52b4e3"
format_stamp: "Formatted at 2026-09-26 20:55:25 on dist-test-slave-2dg3"
server_key: "51205398d2bee1c481f88c212f0a7c15"
server_key_iv: "89545a52e0880eeb1bd05eb9274d3b9a"
server_key_version: "encryptionkey@0"
I20260926 20:55:25.977419 23443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:26.010421 23443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:26.014438 23443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:26.014737 23443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:26.015770 23443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:26.017122 23443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:26.017194 23443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:26.017338 23443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:26.017395 23443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:26.032073 23303 heartbeater.cc:503] Master 127.8.249.190:40915 was elected leader, sending a full tablet report...
I20260926 20:55:26.051879 23443 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:39267
I20260926 20:55:26.051915 23570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:39267 every 8 connection(s)
I20260926 20:55:26.053081 23443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:55:26.057652 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 23443
I20260926 20:55:26.057899 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:55:26.058351 9190 external_mini_cluster.cc:1428] Setting key 7b0a79b2f894cbeeabd2a60b0520563f
Sep 26 20:55:26 dist-test-slave-2dg3 krb5kdc[23069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456125, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:26.072669 23573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.055744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:34901 (local address 127.8.249.190:40915)
0926 20:55:26.056187 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:26.056192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:26.057166 (+ 974us) server_negotiation.cc:407] Connection header received
0926 20:55:26.059210 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:26.059216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:26.059325 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:26.059445 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:26.060873 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.061647 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.063234 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.063626 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.063757 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.067132 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:26.067137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:26.067141 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:26.067174 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:26.069611 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:26.070320 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:26.070330 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:26.070333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:26.070448 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:26.070908 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:26.070913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:26.070916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:26.071188 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:26.071366 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:26.072034 (+ 668us) server_negotiation.cc:299] Negotiation successful
0926 20:55:26.072261 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":212,"threads_started":1}
I20260926 20:55:26.073836 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.056298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:34901)
0926 20:55:26.056996 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:26.057044 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:26.058920 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:26.059688 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:26.059696 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:26.060183 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:26.060713 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:26.060730 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.061808 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.061812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:26.063096 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:26.063107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.063839 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.063848 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:26.064093 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.064712 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:26.064733 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:26.066918 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:26.069752 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:26.069760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:26.069777 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:26.070190 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:26.070568 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:26.070574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:26.070578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:26.070764 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:26.071326 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:26.071335 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:26.071681 (+ 346us) client_negotiation.cc:769] Sending connection context
0926 20:55:26.071946 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 20:55:26.072505 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":273,"threads_started":1}
I20260926 20:55:26.075214 23571 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40915
I20260926 20:55:26.075537 23571 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:26.076256 23571 heartbeater.cc:511] Master 127.8.249.190:40915 requested a full tablet report, sending...
I20260926 20:55:26.077741 23114 ts_manager.cc:194] Registered new tserver with Master: 7132b5eaf5ba4112ae5063e93d52b4e3 (127.8.249.131:39267)
I20260926 20:55:26.078550 23114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:34901
I20260926 20:55:26.086022 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:55:26.102744 23573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.090677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.8.249.190:40915)
0926 20:55:26.090908 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:26.090913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:26.091311 (+ 398us) server_negotiation.cc:407] Connection header received
0926 20:55:26.091458 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:26.091463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:26.091533 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:26.091619 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:26.092982 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.093674 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.095084 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.095470 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.095570 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.097820 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:26.097825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:26.097828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:26.097860 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:26.100514 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:26.101167 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:26.101171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:26.101173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:26.101289 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:26.101587 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:26.101590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:26.101592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:26.101888 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:26.102062 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:26.102342 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 20:55:26.102517 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:55:26.107174 23114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:55:26.114147 23114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:55:26.142171 23586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.127211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.8.249.130:35155)
0926 20:55:26.128789 (+ 1578us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:26.128803 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:26.130747 (+ 1944us) server_negotiation.cc:407] Connection header received
0926 20:55:26.131095 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:26.131107 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:26.131413 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:26.131617 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:26.132687 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.136060 (+ 3373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.139436 (+ 3376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.140933 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.141066 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.141289 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:26.141645 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 20:55:26.141815 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1365,"thread_start_us":1227,"threads_started":1}
I20260926 20:55:26.142205 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.126758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:35155 (local address 127.0.0.1:46408)
0926 20:55:26.129343 (+ 2585us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:26.129378 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:26.129563 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:26.131940 (+ 2377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:26.131951 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:26.131990 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:26.132440 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:26.132456 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.136269 (+ 3813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.136278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:26.139238 (+ 2960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:26.139251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.141179 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.141186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:26.141402 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.141443 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 20:55:26.141600 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 20:55:26.141810 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2365,"thread_start_us":2187,"threads_started":1}
I20260926 20:55:26.145999 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.126824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.8.249.129:33765)
0926 20:55:26.137738 (+ 10914us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:26.137747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:26.137775 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:55:26.137871 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:26.137880 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:26.138125 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:26.138306 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:26.139319 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.140897 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.143795 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.145076 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.145183 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.145336 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:26.145643 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 20:55:26.145765 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10733,"thread_start_us":10538,"threads_started":1}
I20260926 20:55:26.146063 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.126763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:33765 (local address 127.0.0.1:34786)
0926 20:55:26.136418 (+ 9655us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:26.136439 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:26.136582 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:26.138560 (+ 1978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:26.138565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:26.138590 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:26.139114 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:26.139130 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.141049 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.141054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:26.143641 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:26.143654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.145284 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.145290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:26.145459 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.145493 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:55:26.145579 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 20:55:26.145699 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9490,"mutex_wait_us":1029,"thread_start_us":8572,"threads_started":1}
I20260926 20:55:26.146095 23372 tablet_service.cc:1467] Processing CreateTablet for tablet d095afa6400a45938ea49e78d3e20224 (DEFAULT_TABLE table=test-table [id=63bcbe26428a43dcb72a338deb971317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:26.147610 23372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d095afa6400a45938ea49e78d3e20224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:26.148980 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.130061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.8.249.131:39267)
0926 20:55:26.136821 (+ 6760us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:26.136835 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:26.137653 (+ 818us) server_negotiation.cc:407] Connection header received
0926 20:55:26.137772 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:26.137780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:26.138066 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:26.138267 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:26.139364 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.140385 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.143332 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.147684 (+ 4352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.147814 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.148027 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:26.148432 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 20:55:26.148591 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6553,"thread_start_us":6402,"threads_started":1}
I20260926 20:55:26.149441 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.130333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:39267 (local address 127.0.0.1:44392)
0926 20:55:26.137420 (+ 7087us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:26.137437 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:26.137555 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:26.138571 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:26.138578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:26.138604 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:26.139165 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:26.139177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.140541 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.140547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:26.143146 (+ 2599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:26.143161 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.147982 (+ 4821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.147989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:26.148186 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.148217 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 20:55:26.148303 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 20:55:26.148767 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6924,"mutex_wait_us":3,"thread_start_us":6735,"threads_started":1}
I20260926 20:55:26.149755 23238 tablet_service.cc:1467] Processing CreateTablet for tablet d095afa6400a45938ea49e78d3e20224 (DEFAULT_TABLE table=test-table [id=63bcbe26428a43dcb72a338deb971317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:26.151144 23238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d095afa6400a45938ea49e78d3e20224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:26.153273 23506 tablet_service.cc:1467] Processing CreateTablet for tablet d095afa6400a45938ea49e78d3e20224 (DEFAULT_TABLE table=test-table [id=63bcbe26428a43dcb72a338deb971317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:26.154686 23506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d095afa6400a45938ea49e78d3e20224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:26.162315 23590 tablet_bootstrap.cc:492] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724: Bootstrap starting.
I20260926 20:55:26.166100 23591 tablet_bootstrap.cc:492] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a: Bootstrap starting.
I20260926 20:55:26.167234 23590 tablet_bootstrap.cc:654] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:26.168689 23590 log.cc:824] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:26.169927 23591 tablet_bootstrap.cc:654] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:26.170825 23593 tablet_bootstrap.cc:492] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3: Bootstrap starting.
I20260926 20:55:26.171409 23591 log.cc:824] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:26.174336 23591 tablet_bootstrap.cc:492] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a: No bootstrap required, opened a new log
I20260926 20:55:26.174360 23593 tablet_bootstrap.cc:654] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:26.174629 23591 ts_tablet_manager.cc:1397] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 20:55:26.174918 23590 tablet_bootstrap.cc:492] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724: No bootstrap required, opened a new log
I20260926 20:55:26.175176 23590 ts_tablet_manager.cc:1397] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260926 20:55:26.175549 23593 log.cc:824] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:26.180002 23593 tablet_bootstrap.cc:492] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3: No bootstrap required, opened a new log
I20260926 20:55:26.180255 23593 ts_tablet_manager.cc:1397] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 20:55:26.183218 23591 raft_consensus.cc:348] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } } peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } }
I20260926 20:55:26.183599 23591 raft_consensus.cc:374] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:26.183715 23591 raft_consensus.cc:729] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d992231bfaab4bf7a329e37633a4a30a, State: Initialized, Role: FOLLOWER
I20260926 20:55:26.184337 23591 consensus_queue.cc:260] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } } peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } }
I20260926 20:55:26.185451 23437 heartbeater.cc:503] Master 127.8.249.190:40915 was elected leader, sending a full tablet report...
I20260926 20:55:26.185837 23590 raft_consensus.cc:348] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } } peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } }
I20260926 20:55:26.186275 23590 raft_consensus.cc:374] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:26.186372 23590 raft_consensus.cc:729] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4fae9e9222c48cbaf2c6e34450ee724, State: Initialized, Role: FOLLOWER
I20260926 20:55:26.187011 23590 consensus_queue.cc:260] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } } peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } }
I20260926 20:55:26.187587 23591 ts_tablet_manager.cc:1428] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 20:55:26.189257 23590 ts_tablet_manager.cc:1428] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260926 20:55:26.192445 23593 raft_consensus.cc:348] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } } peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } }
I20260926 20:55:26.192818 23593 raft_consensus.cc:374] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:26.192939 23593 raft_consensus.cc:729] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7132b5eaf5ba4112ae5063e93d52b4e3, State: Initialized, Role: FOLLOWER
I20260926 20:55:26.193454 23593 consensus_queue.cc:260] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } } peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } }
I20260926 20:55:26.194442 23571 heartbeater.cc:503] Master 127.8.249.190:40915 was elected leader, sending a full tablet report...
I20260926 20:55:26.195116 23593 ts_tablet_manager.cc:1428] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
W20260926 20:55:26.254056 23304 tablet.cc:2369] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:55:26.277339 23438 tablet.cc:2369] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:55:26.304733 23572 tablet.cc:2369] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:26.489650 23598 raft_consensus.cc:482] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:55:26.489971 23598 raft_consensus.cc:504] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } } peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } }
I20260926 20:55:26.491726 23598 leader_election.cc:288] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4fae9e9222c48cbaf2c6e34450ee724 (127.8.249.129:33765), d992231bfaab4bf7a329e37633a4a30a (127.8.249.130:35155)
I20260926 20:55:26.497406 23586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.492417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:34205 (local address 127.8.249.130:35155)
0926 20:55:26.492626 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:26.492634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:26.492652 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:55:26.492729 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:26.492734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:26.492803 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:26.492891 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:26.493398 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.494115 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.495767 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.496531 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.496699 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.496775 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:26.497026 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 20:55:26.497147 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:55:26.497416 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.491987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:35155 (local address 127.8.249.131:34205)
0926 20:55:26.492262 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:26.492285 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:26.492410 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:26.493075 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:26.493082 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:26.493110 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:26.493261 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:26.493268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.494260 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.494266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:26.495603 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:26.495614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.496687 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.496693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:26.496872 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.496896 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:55:26.497000 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:55:26.497138 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260926 20:55:26.498200 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.492095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:33765 (local address 127.8.249.131:39137)
0926 20:55:26.492918 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:26.492935 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:26.493038 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:26.493621 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:26.493625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:26.493648 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:26.493984 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:26.493993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.495007 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.495011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:26.496295 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:26.496307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.497440 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.497446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:26.497644 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.497682 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 20:55:26.497780 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:55:26.497920 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":522,"threads_started":1}
I20260926 20:55:26.498206 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.493138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:39137 (local address 127.8.249.129:33765)
0926 20:55:26.493341 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:26.493345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:26.493368 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:55:26.493427 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:26.493430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:26.493486 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:26.493604 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:26.494142 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.494842 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.496462 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.497235 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.497428 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.497495 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:26.497825 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 20:55:26.497935 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:55:26.498564 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d095afa6400a45938ea49e78d3e20224" candidate_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d992231bfaab4bf7a329e37633a4a30a" is_pre_election: true
I20260926 20:55:26.499176 23392 raft_consensus.cc:2454] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7132b5eaf5ba4112ae5063e93d52b4e3 in term 0.
I20260926 20:55:26.499214 23258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d095afa6400a45938ea49e78d3e20224" candidate_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" is_pre_election: true
I20260926 20:55:26.499770 23258 raft_consensus.cc:2454] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7132b5eaf5ba4112ae5063e93d52b4e3 in term 0.
I20260926 20:55:26.499830 23459 leader_election.cc:302] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7132b5eaf5ba4112ae5063e93d52b4e3, d992231bfaab4bf7a329e37633a4a30a; no voters:
I20260926 20:55:26.500222 23598 raft_consensus.cc:2781] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:55:26.500393 23598 raft_consensus.cc:482] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:55:26.500505 23598 raft_consensus.cc:3037] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:26.502703 23598 raft_consensus.cc:504] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } } peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } }
I20260926 20:55:26.503369 23598 leader_election.cc:288] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [CANDIDATE]: Term 1 election: Requested vote from peers a4fae9e9222c48cbaf2c6e34450ee724 (127.8.249.129:33765), d992231bfaab4bf7a329e37633a4a30a (127.8.249.130:35155)
I20260926 20:55:26.503713 23258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d095afa6400a45938ea49e78d3e20224" candidate_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4fae9e9222c48cbaf2c6e34450ee724"
I20260926 20:55:26.503834 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d095afa6400a45938ea49e78d3e20224" candidate_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d992231bfaab4bf7a329e37633a4a30a"
I20260926 20:55:26.503918 23258 raft_consensus.cc:3037] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:26.504074 23392 raft_consensus.cc:3037] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:26.506184 23392 raft_consensus.cc:2454] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7132b5eaf5ba4112ae5063e93d52b4e3 in term 1.
I20260926 20:55:26.506206 23258 raft_consensus.cc:2454] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7132b5eaf5ba4112ae5063e93d52b4e3 in term 1.
I20260926 20:55:26.506611 23459 leader_election.cc:302] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7132b5eaf5ba4112ae5063e93d52b4e3, d992231bfaab4bf7a329e37633a4a30a; no voters:
I20260926 20:55:26.506916 23598 raft_consensus.cc:2781] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:26.507311 23598 raft_consensus.cc:686] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 1 LEADER]: Becoming Leader. State: Replica: 7132b5eaf5ba4112ae5063e93d52b4e3, State: Running, Role: LEADER
I20260926 20:55:26.508014 23598 consensus_queue.cc:237] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } } peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } }
I20260926 20:55:26.512331 23113 catalog_manager.cc:5495] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7132b5eaf5ba4112ae5063e93d52b4e3 (127.8.249.131). New cstate: current_term: 1 leader_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } health_report { overall_health: HEALTHY } } }
I20260926 20:55:26.594316 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.589059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.8.249.131:39267)
0926 20:55:26.589270 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:26.589279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:26.589298 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:55:26.589389 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:26.589393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:26.589477 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:26.589626 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:26.590001 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.590713 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.592227 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.592632 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.592766 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.593239 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:26.593716 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:26.593876 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:26.593979 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 20:55:26.594070 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:55:26.602175 23258 raft_consensus.cc:1261] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724 [term 1 FOLLOWER]: Refusing update from remote peer 7132b5eaf5ba4112ae5063e93d52b4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:26.602186 23392 raft_consensus.cc:1261] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a [term 1 FOLLOWER]: Refusing update from remote peer 7132b5eaf5ba4112ae5063e93d52b4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:26.603164 23600 consensus_queue.cc:1035] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:26.603513 23598 consensus_queue.cc:1035] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:26.614719 23603 mvcc.cc:205] Tried to move back new op lower bound from 7333708294553026560 to 7333708294180745216. Current Snapshot: MvccSnapshot[applied={T|T < 7333708294553026560}]
I20260926 20:55:26.615950 23605 mvcc.cc:205] Tried to move back new op lower bound from 7333708294553026560 to 7333708294180745216. Current Snapshot: MvccSnapshot[applied={T|T < 7333708294553026560}]
I20260926 20:55:26.619086 23604 mvcc.cc:205] Tried to move back new op lower bound from 7333708294553026560 to 7333708294180745216. Current Snapshot: MvccSnapshot[applied={T|T < 7333708294553026560}]
I20260926 20:55:26.628301 23113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:55:26.628616 23113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:55:26.632320 23113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Sending DeleteTablet for 3 replicas of tablet d095afa6400a45938ea49e78d3e20224
I20260926 20:55:26.633514 23238 tablet_service.cc:1514] Processing DeleteTablet for tablet d095afa6400a45938ea49e78d3e20224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:26 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:34786
I20260926 20:55:26.633590 23372 tablet_service.cc:1514] Processing DeleteTablet for tablet d095afa6400a45938ea49e78d3e20224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:26 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:46408
I20260926 20:55:26.633673 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 23175
I20260926 20:55:26.634410 23506 tablet_service.cc:1514] Processing DeleteTablet for tablet d095afa6400a45938ea49e78d3e20224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:26 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:44392
I20260926 20:55:26.647423 23613 tablet_replica.cc:330] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724: stopping tablet replica
I20260926 20:55:26.647971 23613 raft_consensus.cc:2229] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:55:26.649969 23613 raft_consensus.cc:2258] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:26.650250 23614 tablet_replica.cc:330] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a: stopping tablet replica
I20260926 20:55:26.650782 23614 raft_consensus.cc:2229] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:55:26.651382 23614 raft_consensus.cc:2258] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:26.651574 23616 tablet_replica.cc:330] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3: stopping tablet replica
I20260926 20:55:26.652674 23613 ts_tablet_manager.cc:1905] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:26.654181 23616 raft_consensus.cc:2229] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:55:26.654901 23616 raft_consensus.cc:2258] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:26.655396 23614 ts_tablet_manager.cc:1905] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:26.660362 23613 ts_tablet_manager.cc:1918] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:26.660563 23613 log.cc:1196] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/d095afa6400a45938ea49e78d3e20224
I20260926 20:55:26.661131 23613 ts_tablet_manager.cc:1939] T d095afa6400a45938ea49e78d3e20224 P a4fae9e9222c48cbaf2c6e34450ee724: Deleting consensus metadata
Sep 26 20:55:26 dist-test-slave-2dg3 krb5kdc[23069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:55:26.663255 23614 ts_tablet_manager.cc:1918] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:26.663247 23100 catalog_manager.cc:4841] TS a4fae9e9222c48cbaf2c6e34450ee724 (127.8.249.129:33765): tablet d095afa6400a45938ea49e78d3e20224 (table test-table [id=63bcbe26428a43dcb72a338deb971317]) successfully deleted
I20260926 20:55:26.663482 23614 log.cc:1196] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/d095afa6400a45938ea49e78d3e20224
I20260926 20:55:26.664098 23614 ts_tablet_manager.cc:1939] T d095afa6400a45938ea49e78d3e20224 P d992231bfaab4bf7a329e37633a4a30a: Deleting consensus metadata
I20260926 20:55:26.665017 23616 ts_tablet_manager.cc:1905] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:26.666064 23098 catalog_manager.cc:4841] TS d992231bfaab4bf7a329e37633a4a30a (127.8.249.130:35155): tablet d095afa6400a45938ea49e78d3e20224 (table test-table [id=63bcbe26428a43dcb72a338deb971317]) successfully deleted
I20260926 20:55:26.673319 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.634957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.8.249.129:33765)
0926 20:55:26.635154 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:26.635160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:26.650564 (+ 15404us) server_negotiation.cc:407] Connection header received
0926 20:55:26.651001 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:26.651008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:26.651122 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:26.651234 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:26.653559 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.654753 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.656958 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.657667 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.657786 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.665029 (+ 7243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:26.665044 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:26.665064 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:26.665112 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:26.669429 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:26.670558 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:26.670569 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:26.670573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:26.670776 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:26.671321 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:26.671339 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:26.671344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:26.671874 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:26.672191 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:26.672626 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 20:55:26.672877 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 20:55:26.673744 23616 ts_tablet_manager.cc:1918] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:26.673962 23616 log.cc:1196] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/d095afa6400a45938ea49e78d3e20224
I20260926 20:55:26.674263 23298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:26.674669 23616 ts_tablet_manager.cc:1939] T d095afa6400a45938ea49e78d3e20224 P 7132b5eaf5ba4112ae5063e93d52b4e3: Deleting consensus metadata
I20260926 20:55:26.677806 23098 catalog_manager.cc:4841] TS 7132b5eaf5ba4112ae5063e93d52b4e3 (127.8.249.131:39267): tablet d095afa6400a45938ea49e78d3e20224 (table test-table [id=63bcbe26428a43dcb72a338deb971317]) successfully deleted
I20260926 20:55:26.765295 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 23175
I20260926 20:55:26.789430 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 23309
Sep 26 20:55:26 dist-test-slave-2dg3 krb5kdc[23069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:55:26.807299 23586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.790225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.8.249.130:35155)
0926 20:55:26.790392 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:26.790397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:26.790470 (+ 73us) server_negotiation.cc:407] Connection header received
0926 20:55:26.790657 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:26.790661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:26.790734 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:26.790876 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:26.792501 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.793218 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.795226 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.795609 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.795705 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.801317 (+ 5612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:26.801334 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:26.801351 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:26.801393 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:26.804331 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:26.805265 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:26.805272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:26.805275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:26.805414 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:26.805810 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:26.805813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:26.805815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:26.806176 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:26.806387 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:26.806747 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 20:55:26.806982 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:55:26.807925 23432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:26.898852 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 23309
I20260926 20:55:26.921801 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 23443
Sep 26 20:55:26 dist-test-slave-2dg3 krb5kdc[23069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:55:26.940696 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:26.922620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44418 (local address 127.8.249.131:39267)
0926 20:55:26.922772 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:26.922776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:26.922873 (+ 97us) server_negotiation.cc:407] Connection header received
0926 20:55:26.923068 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:26.923072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:26.923144 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:26.923224 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:26.924981 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.925787 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.927775 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:26.928225 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:26.928318 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:26.933593 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:26.933608 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:26.933628 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:26.933677 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:26.937805 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:26.938475 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:26.938485 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:26.938489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:26.938665 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:26.939095 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:26.939106 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:26.939110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:26.939578 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:26.939895 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:26.940069 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 20:55:26.940282 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:55:26.941505 23566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:27.065912 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 23443
I20260926 20:55:27.085707 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 23082
I20260926 20:55:27.086884 23144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:27.197577 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 23082
I20260926 20:55:27.224304 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:40915
--webserver_interface=127.8.249.190
--webserver_port=46801
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:40915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:27.557690 23621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:27.558099 23621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:27.558270 23621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:27.565824 23621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:27.566004 23621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:27.566133 23621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:27.566238 23621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:27.566336 23621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:27.577570 23621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:40915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:40915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=46801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:27.579041 23621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:27.580420 23621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:27.603695 23628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:27.603876 23630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:27.603732 23627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:27.604633 23621 server_base.cc:1034] running on GCE node
I20260926 20:55:27.605522 23621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:27.607298 23621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:27.608568 23621 hybrid_clock.cc:648] HybridClock initialized: now 1790456127608518 us; error 39 us; skew 500 ppm
I20260926 20:55:27.608990 23621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:27 dist-test-slave-2dg3 krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456127, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:27.612594 23621 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:55:27.614203 23621 webserver.cc:460] Webserver started at http://127.8.249.190:46801/ using document root <none> and password file <none>
I20260926 20:55:27.614984 23621 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:27.615113 23621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:27.620993 23621 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 20:55:27.623986 23637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:27.625388 23621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:55:27.625543 23621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "c3d47f285d274861a613789d7fbcf84f"
format_stamp: "Formatted at 2026-09-26 20:55:24 on dist-test-slave-2dg3"
server_key: "5d6892167ff953a4dfd5804ac9795003"
server_key_iv: "2b651e41d6f43b6a869e5a3467279dd4"
server_key_version: "encryptionkey@0"
I20260926 20:55:27.626560 23621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:27.646940 23621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:27.651189 23621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:27.651432 23621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:27.672719 23621 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:40915
I20260926 20:55:27.672736 23688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:40915 every 8 connection(s)
I20260926 20:55:27.674650 23621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:27.681892 23689 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:55:27.682662 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 23621
I20260926 20:55:27.683583 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:33765
--local_ip_for_outbound_sockets=127.8.249.129
--tserver_master_addrs=127.8.249.190:40915
--webserver_port=37165
--webserver_interface=127.8.249.129
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:55:27.685540 23689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Bootstrap starting.
I20260926 20:55:27.717311 23689 log.cc:824] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:27.733379 23689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:55:27.734061 23689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Bootstrap complete.
I20260926 20:55:27.748279 23689 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } }
I20260926 20:55:27.749505 23689 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d47f285d274861a613789d7fbcf84f, State: Initialized, Role: FOLLOWER
I20260926 20:55:27.750311 23689 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } }
I20260926 20:55:27.750679 23689 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:55:27.750872 23689 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:55:27.751108 23689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:55:27.757824 23689 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } }
I20260926 20:55:27.758541 23689 leader_election.cc:302] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3d47f285d274861a613789d7fbcf84f; no voters:
I20260926 20:55:27.759227 23689 leader_election.cc:288] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:55:27.759409 23693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:55:27.760936 23693 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 2 LEADER]: Becoming Leader. State: Replica: c3d47f285d274861a613789d7fbcf84f, State: Running, Role: LEADER
I20260926 20:55:27.761539 23689 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:27.761474 23693 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } }
I20260926 20:55:27.764292 23694 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c3d47f285d274861a613789d7fbcf84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } } }
I20260926 20:55:27.764729 23694 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:27.764261 23695 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3d47f285d274861a613789d7fbcf84f. Latest consensus state: current_term: 2 leader_uuid: "c3d47f285d274861a613789d7fbcf84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } } }
I20260926 20:55:27.764938 23695 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:27.779548 23699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:27.790581 23699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:27.792048 23699 catalog_manager.cc:1259] Loaded cluster ID: 3e4c427df9f34a34bb598376c69c56a1
I20260926 20:55:27.792145 23699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:55:27.795185 23699 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 20:55:27.813277 23699 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 20:55:27.834718 23699 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:27.836433 23699 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Loaded TSK: 0
I20260926 20:55:27.837915 23699 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:55:28.065416 23691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:28.065694 23691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:28.065867 23691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:28.073238 23691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:28.073393 23691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:28.073658 23691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:55:28.083621 23691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:33765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=37165
--tserver_master_addrs=127.8.249.190:40915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:28.084982 23691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:28.086247 23691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:28.101099 23716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:28.100987 23717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:28.101492 23719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:28.101996 23691 server_base.cc:1034] running on GCE node
I20260926 20:55:28.102670 23691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:28.103693 23691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:28.104918 23691 hybrid_clock.cc:648] HybridClock initialized: now 1790456128104852 us; error 58 us; skew 500 ppm
I20260926 20:55:28.105266 23691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:28 dist-test-slave-2dg3 krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456128, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:28.108623 23691 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:55:28.110221 23691 webserver.cc:460] Webserver started at http://127.8.249.129:37165/ using document root <none> and password file <none>
I20260926 20:55:28.110988 23691 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:28.111145 23691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:28.116585 23691 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:55:28.119454 23726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:28.120854 23691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:55:28.121026 23691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "a4fae9e9222c48cbaf2c6e34450ee724"
format_stamp: "Formatted at 2026-09-26 20:55:24 on dist-test-slave-2dg3"
server_key: "f44c777d5957e8ded95e50b7502ea58c"
server_key_iv: "ebb80d8037e3fd3d08d6e723499bf646"
server_key_version: "encryptionkey@0"
I20260926 20:55:28.121978 23691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:28.135687 23691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:28.140142 23691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:28.140441 23691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:28.141537 23691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:28.143002 23691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:28.143117 23691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:28.143250 23691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:28.143345 23691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:28.179128 23691 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:33765
I20260926 20:55:28.179172 23838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:33765 every 8 connection(s)
I20260926 20:55:28.180603 23691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:55:28.182711 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 23691
I20260926 20:55:28.183540 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:35155
--local_ip_for_outbound_sockets=127.8.249.130
--tserver_master_addrs=127.8.249.190:40915
--webserver_port=39071
--webserver_interface=127.8.249.130
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:55:28 dist-test-slave-2dg3 krb5kdc[23069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456128, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:28.214740 23842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:28.183962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:38061)
0926 20:55:28.184975 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:28.185041 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:28.188091 (+ 3050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:28.194348 (+ 6257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:28.194363 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:28.195208 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:28.196035 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:28.196066 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:28.197654 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:28.197662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:28.199832 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:28.199849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:28.200904 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:28.200913 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:28.201234 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:28.202004 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:28.202039 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:28.205405 (+ 3366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:28.210604 (+ 5199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:28.210612 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:28.210625 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:28.211168 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:28.211685 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:28.211688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:28.211691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:28.211884 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:28.212787 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:28.212793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:28.213103 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 20:55:28.213329 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 20:55:28.213736 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":354,"threads_started":1}
I20260926 20:55:28.215570 23841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:28.183635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:38061 (local address 127.8.249.190:40915)
0926 20:55:28.193024 (+ 9389us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:28.193037 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:28.193081 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:55:28.193415 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:28.193430 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:28.193787 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:28.194106 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:28.196280 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:28.197471 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:28.199989 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:28.200669 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:28.200801 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:28.205768 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:28.205780 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:28.205803 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:28.205860 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:28.210424 (+ 4564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:28.211339 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:28.211350 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:28.211359 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:28.211571 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:28.212000 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:28.212006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:28.212010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:28.212651 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:28.212845 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:28.213594 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 20:55:28.214016 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9162,"thread_start_us":8999,"threads_started":1}
I20260926 20:55:28.218189 23839 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40915
I20260926 20:55:28.218495 23839 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:28.219319 23839 heartbeater.cc:511] Master 127.8.249.190:40915 requested a full tablet report, sending...
I20260926 20:55:28.221392 23654 ts_manager.cc:194] Registered new tserver with Master: a4fae9e9222c48cbaf2c6e34450ee724 (127.8.249.129:33765)
I20260926 20:55:28.222904 23654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:38061
W20260926 20:55:28.512625 23843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:28.512961 23843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:28.513134 23843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:28.519938 23843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:28.520066 23843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:28.520244 23843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:55:28.529804 23843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:35155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=39071
--tserver_master_addrs=127.8.249.190:40915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:28.531105 23843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:28.532398 23843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:28.543738 23848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:28.544767 23849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:28.545637 23851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:28.546253 23843 server_base.cc:1034] running on GCE node
I20260926 20:55:28.546959 23843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:28.547919 23843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:28.549118 23843 hybrid_clock.cc:648] HybridClock initialized: now 1790456128549055 us; error 56 us; skew 500 ppm
I20260926 20:55:28.549439 23843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:28 dist-test-slave-2dg3 krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456128, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:28.552541 23843 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:55:28.553975 23843 webserver.cc:460] Webserver started at http://127.8.249.130:39071/ using document root <none> and password file <none>
I20260926 20:55:28.554695 23843 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:28.554893 23843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:28.560189 23843 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 20:55:28.563004 23858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:28.564304 23843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 20:55:28.564478 23843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "d992231bfaab4bf7a329e37633a4a30a"
format_stamp: "Formatted at 2026-09-26 20:55:25 on dist-test-slave-2dg3"
server_key: "ed62dfab1cce587383a682462666a6b1"
server_key_iv: "099fc7c9c2d6a0129e65c0bdee472198"
server_key_version: "encryptionkey@0"
I20260926 20:55:28.565448 23843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:28.586097 23843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:28.590528 23843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:28.590848 23843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:28.591828 23843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:28.593132 23843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:28.593218 23843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:28.593375 23843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:28.593468 23843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:28.625645 23843 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:35155
I20260926 20:55:28.625674 23970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:35155 every 8 connection(s)
I20260926 20:55:28.627388 23843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:55:28.630141 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 23843
I20260926 20:55:28.630770 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:39267
--local_ip_for_outbound_sockets=127.8.249.131
--tserver_master_addrs=127.8.249.190:40915
--webserver_port=38329
--webserver_interface=127.8.249.131
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:55:28 dist-test-slave-2dg3 krb5kdc[23069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456128, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:28.655238 23841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:28.630100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:56645 (local address 127.8.249.190:40915)
0926 20:55:28.630352 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:28.630359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:28.635796 (+ 5437us) server_negotiation.cc:407] Connection header received
0926 20:55:28.638404 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:28.638411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:28.638510 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:28.638645 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:28.640308 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:28.641479 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:28.643248 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:28.644037 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:28.644180 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:28.647868 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:28.647876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:28.647883 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:28.647928 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:28.651641 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:28.652494 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:28.652500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:28.652506 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:28.652675 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:28.653165 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:28.653173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:28.653179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:28.653576 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:28.653742 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:28.654467 (+ 725us) server_negotiation.cc:299] Negotiation successful
0926 20:55:28.654746 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 20:55:28.656071 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:28.630689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:56645)
0926 20:55:28.635569 (+ 4880us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:28.635626 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:28.638071 (+ 2445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:28.638974 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:28.638985 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:28.639533 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:28.640117 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:28.640135 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:28.641663 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:28.641667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:28.643093 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:28.643107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:28.644253 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:28.644258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:28.644463 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:28.645183 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:28.645211 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:28.647622 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:28.651820 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:28.651847 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:28.651861 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:28.652348 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:28.652813 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:28.652818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:28.652820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:28.653026 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:28.653699 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:28.653705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:28.654050 (+ 345us) client_negotiation.cc:769] Sending connection context
0926 20:55:28.654352 (+ 302us) client_negotiation.cc:240] Negotiation successful
0926 20:55:28.654836 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4437,"thread_start_us":4150,"threads_started":1}
I20260926 20:55:28.657656 23971 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40915
I20260926 20:55:28.658006 23971 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:28.658716 23971 heartbeater.cc:511] Master 127.8.249.190:40915 requested a full tablet report, sending...
I20260926 20:55:28.660154 23654 ts_manager.cc:194] Registered new tserver with Master: d992231bfaab4bf7a329e37633a4a30a (127.8.249.130:35155)
I20260926 20:55:28.660951 23654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:56645
W20260926 20:55:28.965420 23974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:28.965775 23974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:28.965965 23974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:28.973676 23974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:28.973861 23974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:28.974051 23974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:55:28.984407 23974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:39267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=38329
--tserver_master_addrs=127.8.249.190:40915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:28.985803 23974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:28.987294 23974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:29.001349 23979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:29.001438 23982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:29.001423 23980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:29.003033 23974 server_base.cc:1034] running on GCE node
I20260926 20:55:29.003628 23974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:29.004561 23974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:29.005812 23974 hybrid_clock.cc:648] HybridClock initialized: now 1790456129005769 us; error 39 us; skew 500 ppm
I20260926 20:55:29.006214 23974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:29 dist-test-slave-2dg3 krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456129, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:29.009289 23974 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:55:29.010725 23974 webserver.cc:460] Webserver started at http://127.8.249.131:38329/ using document root <none> and password file <none>
I20260926 20:55:29.011459 23974 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:29.011566 23974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:29.016736 23974 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 20:55:29.019544 23989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:29.020865 23974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:55:29.021027 23974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "7132b5eaf5ba4112ae5063e93d52b4e3"
format_stamp: "Formatted at 2026-09-26 20:55:25 on dist-test-slave-2dg3"
server_key: "51205398d2bee1c481f88c212f0a7c15"
server_key_iv: "89545a52e0880eeb1bd05eb9274d3b9a"
server_key_version: "encryptionkey@0"
I20260926 20:55:29.022009 23974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:29.045168 23974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:29.049193 23974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:29.049466 23974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:29.050503 23974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:29.051882 23974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:29.051970 23974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:29.052126 23974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:29.052215 23974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:29.085754 23974 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:39267
I20260926 20:55:29.085846 24101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:39267 every 8 connection(s)
I20260926 20:55:29.087312 23974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:55:29.095856 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 23974
Sep 26 20:55:29 dist-test-slave-2dg3 krb5kdc[23069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456129, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:29.109038 23841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.090065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:54369 (local address 127.8.249.190:40915)
0926 20:55:29.090252 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:29.090256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:29.091477 (+ 1221us) server_negotiation.cc:407] Connection header received
0926 20:55:29.093220 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:29.093225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:29.093299 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:29.093398 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:29.094759 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.095818 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.097783 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.098505 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.098657 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.102625 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:29.102632 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:29.102639 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:29.102679 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:29.105511 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:29.106235 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:29.106242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:29.106250 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:29.106424 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:29.106957 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:29.106968 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:29.106973 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:29.107340 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:29.107490 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:29.108351 (+ 861us) server_negotiation.cc:299] Negotiation successful
0926 20:55:29.108640 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 20:55:29.109913 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.090571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:54369)
0926 20:55:29.091318 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:29.091360 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:29.092962 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:29.093588 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:29.093596 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:29.094082 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:29.094588 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:29.094603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.095974 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.095981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:29.097609 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:29.097620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.098709 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.098714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:29.098976 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.099655 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:29.099682 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:29.102381 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:29.105655 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:29.105664 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:29.105682 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:29.106102 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:29.106546 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:29.106550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:29.106553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:29.106766 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:29.107467 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:29.107473 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:29.107832 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 20:55:29.108213 (+ 381us) client_negotiation.cc:240] Negotiation successful
0926 20:55:29.108639 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":340,"threads_started":1}
I20260926 20:55:29.111603 24102 heartbeater.cc:347] Connected to a master server at 127.8.249.190:40915
I20260926 20:55:29.111934 24102 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:29.112803 24102 heartbeater.cc:511] Master 127.8.249.190:40915 requested a full tablet report, sending...
I20260926 20:55:29.113112 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.096849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.8.249.190:40915)
0926 20:55:29.097284 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:29.097289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:29.097680 (+ 391us) server_negotiation.cc:407] Connection header received
0926 20:55:29.097867 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:29.097870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:29.097929 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:29.098014 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:29.099936 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.101061 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.102662 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.103399 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.103516 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.106407 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:29.106414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:29.106418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:29.106451 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:29.109635 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:29.110919 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:29.110940 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:29.110944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:29.111084 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:29.111546 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:29.111552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:29.111558 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:29.111886 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:29.112151 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:29.112500 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 20:55:29.112727 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":230,"threads_started":1}
I20260926 20:55:29.114354 23654 ts_manager.cc:194] Registered new tserver with Master: 7132b5eaf5ba4112ae5063e93d52b4e3 (127.8.249.131:39267)
I20260926 20:55:29.115502 23654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:54369
I20260926 20:55:29.115778 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:55:29.122476 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.116752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48706 (local address 127.8.249.190:40915)
0926 20:55:29.116911 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:29.116915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:29.117374 (+ 459us) server_negotiation.cc:407] Connection header received
0926 20:55:29.117615 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:29.117619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:29.117737 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:29.117873 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:29.118295 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.119035 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.120456 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.120863 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.121000 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.121494 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:29.121939 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:29.122097 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:29.122154 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 20:55:29.122227 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:55:29.123395 23654 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:55:29.131361 23654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:55:29.158051 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.143568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.8.249.130:35155)
0926 20:55:29.144321 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:29.144334 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:29.145315 (+ 981us) server_negotiation.cc:407] Connection header received
0926 20:55:29.145825 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:29.145838 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:29.146127 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:29.146297 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:29.147610 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.149738 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.153814 (+ 4076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.156129 (+ 2315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.156274 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.156627 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:29.157499 (+ 872us) server_negotiation.cc:299] Negotiation successful
0926 20:55:29.157699 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":360,"threads_started":1}
I20260926 20:55:29.159157 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.144190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:33765 (local address 127.0.0.1:34794)
0926 20:55:29.146258 (+ 2068us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:29.146301 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:29.146444 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:29.147598 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:29.147607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:29.147647 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:29.148197 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:29.148214 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.151038 (+ 2824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.151048 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:29.154463 (+ 3415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:29.154489 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.157732 (+ 3243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.157743 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:29.158052 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.158114 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 20:55:29.158260 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 20:55:29.158580 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1969,"thread_start_us":445,"threads_started":1}
I20260926 20:55:29.159431 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.143175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:35155 (local address 127.0.0.1:46412)
0926 20:55:29.145123 (+ 1948us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:29.145178 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:29.145508 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:29.146755 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:29.146767 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:29.146856 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:29.147377 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:29.147393 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.149971 (+ 2578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.149979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:29.153525 (+ 3546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:29.153551 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.156824 (+ 3273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.156830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:29.157130 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.157214 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 20:55:29.158586 (+ 1372us) client_negotiation.cc:240] Negotiation successful
0926 20:55:29.158718 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1649,"mutex_wait_us":2,"thread_start_us":2841,"threads_started":1}
I20260926 20:55:29.160127 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.144588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.8.249.129:33765)
0926 20:55:29.145965 (+ 1377us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:29.145976 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:29.146603 (+ 627us) server_negotiation.cc:407] Connection header received
0926 20:55:29.146721 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:29.146730 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:29.147004 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:29.147247 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:29.148493 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.150736 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.154762 (+ 4026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.156980 (+ 2218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.157189 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.157470 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:29.159607 (+ 2137us) server_negotiation.cc:299] Negotiation successful
0926 20:55:29.159822 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1179,"thread_start_us":1009,"threads_started":1}
I20260926 20:55:29.163741 23774 tablet_service.cc:1467] Processing CreateTablet for tablet 7372f82b37e14e3f9a31573088af6e24 (DEFAULT_TABLE table=test-table [id=e48b60397163468ea1ce02f309931340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:29.163770 23906 tablet_service.cc:1467] Processing CreateTablet for tablet 7372f82b37e14e3f9a31573088af6e24 (DEFAULT_TABLE table=test-table [id=e48b60397163468ea1ce02f309931340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:29.165051 23906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7372f82b37e14e3f9a31573088af6e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:29.165110 23774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7372f82b37e14e3f9a31573088af6e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:29.167879 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.146377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:39267 (local address 127.0.0.1:44420)
0926 20:55:29.157856 (+ 11479us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:29.157876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:29.158017 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:29.160193 (+ 2176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:29.160198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:29.160232 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:29.160631 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:29.160645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.162568 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.162573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:29.164515 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:29.164548 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.167088 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.167094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:29.167264 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.167309 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:55:29.167412 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:55:29.167553 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11330,"thread_start_us":11728,"threads_started":1}
I20260926 20:55:29.168035 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.146848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44420 (local address 127.8.249.131:39267)
0926 20:55:29.159521 (+ 12673us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:29.159533 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:29.159572 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:55:29.159670 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:29.159678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:29.159956 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:29.160166 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:29.160837 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.162363 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.164721 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.166868 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.167033 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.167291 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:29.167478 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 20:55:29.167659 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12477,"thread_start_us":12408,"threads_started":1}
I20260926 20:55:29.171926 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 7372f82b37e14e3f9a31573088af6e24 (DEFAULT_TABLE table=test-table [id=e48b60397163468ea1ce02f309931340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:29.173344 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7372f82b37e14e3f9a31573088af6e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:29.183211 24119 tablet_bootstrap.cc:492] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a: Bootstrap starting.
I20260926 20:55:29.189450 24118 tablet_bootstrap.cc:492] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724: Bootstrap starting.
I20260926 20:55:29.194079 24119 tablet_bootstrap.cc:654] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:29.194671 24118 tablet_bootstrap.cc:654] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:29.195593 24120 tablet_bootstrap.cc:492] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3: Bootstrap starting.
I20260926 20:55:29.195994 24118 log.cc:824] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:29.196173 24119 log.cc:824] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:29.199168 24118 tablet_bootstrap.cc:492] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724: No bootstrap required, opened a new log
I20260926 20:55:29.199440 24118 ts_tablet_manager.cc:1397] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 20:55:29.199559 24120 tablet_bootstrap.cc:654] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:29.200635 24120 log.cc:824] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:29.203591 24120 tablet_bootstrap.cc:492] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3: No bootstrap required, opened a new log
I20260926 20:55:29.203918 24120 ts_tablet_manager.cc:1397] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 20:55:29.205783 24119 tablet_bootstrap.cc:492] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a: No bootstrap required, opened a new log
I20260926 20:55:29.206077 24119 ts_tablet_manager.cc:1397] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20260926 20:55:29.208072 24118 raft_consensus.cc:348] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } } peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } }
I20260926 20:55:29.208556 24118 raft_consensus.cc:374] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:29.208660 24118 raft_consensus.cc:729] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4fae9e9222c48cbaf2c6e34450ee724, State: Initialized, Role: FOLLOWER
I20260926 20:55:29.209319 24118 consensus_queue.cc:260] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } } peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } }
I20260926 20:55:29.217736 23839 heartbeater.cc:503] Master 127.8.249.190:40915 was elected leader, sending a full tablet report...
I20260926 20:55:29.217581 24120 raft_consensus.cc:348] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } } peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } }
I20260926 20:55:29.218096 24120 raft_consensus.cc:374] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:29.218223 24120 raft_consensus.cc:729] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7132b5eaf5ba4112ae5063e93d52b4e3, State: Initialized, Role: FOLLOWER
I20260926 20:55:29.218981 24120 consensus_queue.cc:260] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } } peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } }
I20260926 20:55:29.219208 24119 raft_consensus.cc:348] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } } peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } }
I20260926 20:55:29.219660 24119 raft_consensus.cc:374] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:29.219756 24119 raft_consensus.cc:729] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d992231bfaab4bf7a329e37633a4a30a, State: Initialized, Role: FOLLOWER
I20260926 20:55:29.220194 24118 ts_tablet_manager.cc:1428] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724: Time spent starting tablet: real 0.021s user 0.008s sys 0.012s
I20260926 20:55:29.220323 24119 consensus_queue.cc:260] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } } peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } }
I20260926 20:55:29.221163 24102 heartbeater.cc:503] Master 127.8.249.190:40915 was elected leader, sending a full tablet report...
I20260926 20:55:29.221761 24120 ts_tablet_manager.cc:1428] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260926 20:55:29.223151 23971 heartbeater.cc:503] Master 127.8.249.190:40915 was elected leader, sending a full tablet report...
I20260926 20:55:29.223776 24119 ts_tablet_manager.cc:1428] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a: Time spent starting tablet: real 0.018s user 0.009s sys 0.009s
W20260926 20:55:29.339154 24103 tablet.cc:2369] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:29.372815 24125 raft_consensus.cc:482] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:55:29.373133 24125 raft_consensus.cc:504] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } } peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } }
I20260926 20:55:29.374778 24125 leader_election.cc:288] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d992231bfaab4bf7a329e37633a4a30a (127.8.249.130:35155), a4fae9e9222c48cbaf2c6e34450ee724 (127.8.249.129:33765)
W20260926 20:55:29.379392 23972 tablet.cc:2369] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:29.381494 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.375083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:35155 (local address 127.8.249.131:39657)
0926 20:55:29.375404 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:29.375425 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:29.375558 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:29.375955 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:29.375960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:29.375995 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:29.376206 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:29.376216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.377496 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.377499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:29.379396 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:29.379410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.380786 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.380793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:29.381000 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.381034 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:55:29.381125 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:55:29.381265 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":1}
I20260926 20:55:29.381548 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.375151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:39657 (local address 127.8.249.130:35155)
0926 20:55:29.375390 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:29.375397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:29.375685 (+ 288us) server_negotiation.cc:407] Connection header received
0926 20:55:29.375771 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:29.375774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:29.375843 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:29.375914 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:29.376354 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.377357 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.379586 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.380638 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.380768 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.380824 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:29.381166 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 20:55:29.381297 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:55:29.382071 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.375203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:33765 (local address 127.8.249.131:54017)
0926 20:55:29.375963 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:29.375988 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:29.376103 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:29.376454 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:29.376457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:29.376502 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:29.376828 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:29.376836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.377965 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.377970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:29.379937 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:29.379947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.381246 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.381252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:29.381442 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.381474 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 20:55:29.381577 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:55:29.381702 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":64,"thread_start_us":434,"threads_started":1}
I20260926 20:55:29.382156 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.375458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:54017 (local address 127.8.249.129:33765)
0926 20:55:29.375663 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:29.375670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:29.376084 (+ 414us) server_negotiation.cc:407] Connection header received
0926 20:55:29.376239 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:29.376242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:29.376344 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:29.376438 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:29.376981 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.377822 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.380116 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.381014 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.381170 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.381235 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:29.381745 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 20:55:29.381903 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:55:29.382609 23926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7372f82b37e14e3f9a31573088af6e24" candidate_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d992231bfaab4bf7a329e37633a4a30a" is_pre_election: true
I20260926 20:55:29.383225 23926 raft_consensus.cc:2454] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7132b5eaf5ba4112ae5063e93d52b4e3 in term 0.
I20260926 20:55:29.383208 23794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7372f82b37e14e3f9a31573088af6e24" candidate_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" is_pre_election: true
I20260926 20:55:29.383755 23794 raft_consensus.cc:2454] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7132b5eaf5ba4112ae5063e93d52b4e3 in term 0.
I20260926 20:55:29.383947 23990 leader_election.cc:302] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7132b5eaf5ba4112ae5063e93d52b4e3, d992231bfaab4bf7a329e37633a4a30a; no voters:
I20260926 20:55:29.384388 24125 raft_consensus.cc:2781] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:55:29.384562 24125 raft_consensus.cc:482] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:55:29.384682 24125 raft_consensus.cc:3037] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:29.386888 24125 raft_consensus.cc:504] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } } peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } }
I20260926 20:55:29.387552 24125 leader_election.cc:288] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [CANDIDATE]: Term 1 election: Requested vote from peers d992231bfaab4bf7a329e37633a4a30a (127.8.249.130:35155), a4fae9e9222c48cbaf2c6e34450ee724 (127.8.249.129:33765)
I20260926 20:55:29.388055 23926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7372f82b37e14e3f9a31573088af6e24" candidate_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d992231bfaab4bf7a329e37633a4a30a"
I20260926 20:55:29.388087 23794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7372f82b37e14e3f9a31573088af6e24" candidate_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4fae9e9222c48cbaf2c6e34450ee724"
I20260926 20:55:29.388281 23926 raft_consensus.cc:3037] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:29.388301 23794 raft_consensus.cc:3037] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:29.391587 23926 raft_consensus.cc:2454] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7132b5eaf5ba4112ae5063e93d52b4e3 in term 1.
I20260926 20:55:29.391593 23794 raft_consensus.cc:2454] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7132b5eaf5ba4112ae5063e93d52b4e3 in term 1.
I20260926 20:55:29.392092 23992 leader_election.cc:302] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7132b5eaf5ba4112ae5063e93d52b4e3, a4fae9e9222c48cbaf2c6e34450ee724; no voters:
I20260926 20:55:29.392467 24125 raft_consensus.cc:2781] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:29.392938 24125 raft_consensus.cc:686] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 1 LEADER]: Becoming Leader. State: Replica: 7132b5eaf5ba4112ae5063e93d52b4e3, State: Running, Role: LEADER
I20260926 20:55:29.393520 24125 consensus_queue.cc:237] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } } peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } }
I20260926 20:55:29.398455 23654 catalog_manager.cc:5495] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7132b5eaf5ba4112ae5063e93d52b4e3 (127.8.249.131). New cstate: current_term: 1 leader_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7132b5eaf5ba4112ae5063e93d52b4e3" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 39267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 } health_report { overall_health: UNKNOWN } } }
I20260926 20:55:29.424645 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.419481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.8.249.131:39267)
0926 20:55:29.419666 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:29.419670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:29.419688 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:55:29.419794 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:29.419799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:29.419888 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:29.420016 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:29.420339 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.421062 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.422568 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.423051 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.423151 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.423612 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:29.424149 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:29.424261 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:29.424356 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 20:55:29.424428 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 20:55:29.433063 23840 tablet.cc:2369] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:29.433399 23794 raft_consensus.cc:1261] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724 [term 1 FOLLOWER]: Refusing update from remote peer 7132b5eaf5ba4112ae5063e93d52b4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:29.433630 23926 raft_consensus.cc:1261] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a [term 1 FOLLOWER]: Refusing update from remote peer 7132b5eaf5ba4112ae5063e93d52b4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:29.434403 24128 consensus_queue.cc:1035] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4fae9e9222c48cbaf2c6e34450ee724" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:29.434931 24125 consensus_queue.cc:1035] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d992231bfaab4bf7a329e37633a4a30a" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 35155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:29.446842 24131 mvcc.cc:205] Tried to move back new op lower bound from 7333708306149048320 to 7333708306000801792. Current Snapshot: MvccSnapshot[applied={T|T < 7333708306149048320}]
I20260926 20:55:29.455408 24132 mvcc.cc:205] Tried to move back new op lower bound from 7333708306149048320 to 7333708306000801792. Current Snapshot: MvccSnapshot[applied={T|T < 7333708306149048320}]
I20260926 20:55:29.462487 23654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:55:29.462772 23654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:55:29.466274 23654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Sending DeleteTablet for 3 replicas of tablet 7372f82b37e14e3f9a31573088af6e24
I20260926 20:55:29.467617 24037 tablet_service.cc:1514] Processing DeleteTablet for tablet 7372f82b37e14e3f9a31573088af6e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:29 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:44420
I20260926 20:55:29.467599 23906 tablet_service.cc:1514] Processing DeleteTablet for tablet 7372f82b37e14e3f9a31573088af6e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:29 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:46412
I20260926 20:55:29.467689 23774 tablet_service.cc:1514] Processing DeleteTablet for tablet 7372f82b37e14e3f9a31573088af6e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:29 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:34794
I20260926 20:55:29.468011 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 23691
I20260926 20:55:29.470083 24142 tablet_replica.cc:330] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724: stopping tablet replica
I20260926 20:55:29.470649 24142 raft_consensus.cc:2229] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:55:29.471309 24142 raft_consensus.cc:2258] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:29.472146 24141 tablet_replica.cc:330] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3: stopping tablet replica
I20260926 20:55:29.472487 24141 raft_consensus.cc:2229] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:55:29.473058 24141 raft_consensus.cc:2258] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:29.475366 24141 ts_tablet_manager.cc:1905] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:29.475605 24140 tablet_replica.cc:330] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a: stopping tablet replica
I20260926 20:55:29.475996 24140 raft_consensus.cc:2229] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:55:29.476495 24140 raft_consensus.cc:2258] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:29.477032 24142 ts_tablet_manager.cc:1905] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:29.481385 24140 ts_tablet_manager.cc:1905] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:29.483644 24141 ts_tablet_manager.cc:1918] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:29.483898 24141 log.cc:1196] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/7372f82b37e14e3f9a31573088af6e24
I20260926 20:55:29.484014 24142 ts_tablet_manager.cc:1918] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:29.484177 24142 log.cc:1196] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/7372f82b37e14e3f9a31573088af6e24
I20260926 20:55:29.484807 24142 ts_tablet_manager.cc:1939] T 7372f82b37e14e3f9a31573088af6e24 P a4fae9e9222c48cbaf2c6e34450ee724: Deleting consensus metadata
I20260926 20:55:29.484915 24141 ts_tablet_manager.cc:1939] T 7372f82b37e14e3f9a31573088af6e24 P 7132b5eaf5ba4112ae5063e93d52b4e3: Deleting consensus metadata
I20260926 20:55:29.486759 23638 catalog_manager.cc:4841] TS 7132b5eaf5ba4112ae5063e93d52b4e3 (127.8.249.131:39267): tablet 7372f82b37e14e3f9a31573088af6e24 (table test-table [id=e48b60397163468ea1ce02f309931340]) successfully deleted
I20260926 20:55:29.486811 23640 catalog_manager.cc:4841] TS a4fae9e9222c48cbaf2c6e34450ee724 (127.8.249.129:33765): tablet 7372f82b37e14e3f9a31573088af6e24 (table test-table [id=e48b60397163468ea1ce02f309931340]) successfully deleted
I20260926 20:55:29.491006 24140 ts_tablet_manager.cc:1918] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:29.491176 24140 log.cc:1196] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/7372f82b37e14e3f9a31573088af6e24
I20260926 20:55:29.491775 24140 ts_tablet_manager.cc:1939] T 7372f82b37e14e3f9a31573088af6e24 P d992231bfaab4bf7a329e37633a4a30a: Deleting consensus metadata
I20260926 20:55:29.493553 23638 catalog_manager.cc:4841] TS d992231bfaab4bf7a329e37633a4a30a (127.8.249.130:35155): tablet 7372f82b37e14e3f9a31573088af6e24 (table test-table [id=e48b60397163468ea1ce02f309931340]) successfully deleted
I20260926 20:55:29.494618 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.468922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.8.249.129:33765)
0926 20:55:29.469185 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:29.469195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:29.469239 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:55:29.469454 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:29.469462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:29.469570 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:29.469718 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:29.472383 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.473925 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.477364 (+ 3439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.478068 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.478207 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.487263 (+ 9056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:29.487278 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:29.487298 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:29.487351 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:29.491295 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:29.492285 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:29.492295 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:29.492299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:29.492454 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:29.492888 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:29.492893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:29.492897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:29.493369 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:29.493724 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:29.494029 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 20:55:29.494254 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 20:55:29.495280 23834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:29.604593 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 23691
I20260926 20:55:29.628386 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 23843
I20260926 20:55:29.646535 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.629301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.8.249.130:35155)
0926 20:55:29.629481 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:29.629487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:29.629509 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:55:29.629703 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:29.629709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:29.629792 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:29.629918 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:29.631791 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.633020 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.634997 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.635683 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.635815 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.639095 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:29.639110 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:29.639130 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:29.639173 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:29.643239 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:29.644103 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:29.644113 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:29.644117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:29.644271 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:29.644735 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:29.644740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:29.644744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:29.645238 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:29.645531 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:29.645908 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 20:55:29.646135 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:55:29.647536 23966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:29.740149 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 23843
I20260926 20:55:29.757443 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 23974
I20260926 20:55:29.776932 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:29.758267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.8.249.131:39267)
0926 20:55:29.758430 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:29.758434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:29.758456 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:55:29.758657 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:29.758661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:29.758721 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:29.758869 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:29.760637 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.761437 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.763649 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:29.764086 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:29.764185 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:29.770751 (+ 6566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:29.770764 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:29.770781 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:29.770888 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:29.773769 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:29.774513 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:29.774520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:29.774522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:29.774672 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:29.775178 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:29.775184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:29.775188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:29.775681 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:29.776002 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:29.776239 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 20:55:29.776468 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:55:29.777714 24097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:29.915740 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 23974
I20260926 20:55:29.934825 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 23621
I20260926 20:55:29.935909 23684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:30.049808 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 23621
I20260926 20:55:30.071342 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:40915
--webserver_interface=127.8.249.190
--webserver_port=46801
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:40915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:30.397102 24147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:30.397434 24147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:30.397608 24147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:30.404996 24147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:30.405104 24147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:30.405150 24147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:30.405207 24147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:30.405251 24147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:30.415226 24147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:40915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:40915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=46801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:30.416487 24147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:30.417721 24147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:30.428812 24152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:30.428876 24153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:30.429008 24155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:30.429757 24147 server_base.cc:1034] running on GCE node
I20260926 20:55:30.430480 24147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:30.431861 24147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:30.433179 24147 hybrid_clock.cc:648] HybridClock initialized: now 1790456130433136 us; error 27 us; skew 500 ppm
I20260926 20:55:30.433511 24147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:30 dist-test-slave-2dg3 krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456130, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:30.436518 24147 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:55:30.437870 24147 webserver.cc:460] Webserver started at http://127.8.249.190:46801/ using document root <none> and password file <none>
I20260926 20:55:30.438508 24147 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:30.438653 24147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:30.443941 24147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:55:30.446604 24162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:30.447881 24147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:55:30.448063 24147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "c3d47f285d274861a613789d7fbcf84f"
format_stamp: "Formatted at 2026-09-26 20:55:24 on dist-test-slave-2dg3"
server_key: "5d6892167ff953a4dfd5804ac9795003"
server_key_iv: "2b651e41d6f43b6a869e5a3467279dd4"
server_key_version: "encryptionkey@0"
I20260926 20:55:30.449010 24147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:30.462071 24147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:30.465948 24147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:30.466221 24147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:30.485250 24147 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:40915
I20260926 20:55:30.485275 24213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:40915 every 8 connection(s)
I20260926 20:55:30.486896 24147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:30.493503 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 24147
I20260926 20:55:30.493840 24214 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:55:30.494377 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:33765
--local_ip_for_outbound_sockets=127.8.249.129
--tserver_master_addrs=127.8.249.190:40915
--webserver_port=37165
--webserver_interface=127.8.249.129
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:55:30.497115 24214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Bootstrap starting.
I20260926 20:55:30.517553 24214 log.cc:824] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:30.533183 24214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:55:30.533733 24214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f: Bootstrap complete.
I20260926 20:55:30.543659 24214 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } }
I20260926 20:55:30.544687 24214 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d47f285d274861a613789d7fbcf84f, State: Initialized, Role: FOLLOWER
I20260926 20:55:30.545343 24214 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } }
I20260926 20:55:30.545617 24214 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:55:30.545768 24214 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:55:30.545974 24214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:55:30.550148 24214 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } }
I20260926 20:55:30.550664 24214 leader_election.cc:302] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3d47f285d274861a613789d7fbcf84f; no voters:
I20260926 20:55:30.551424 24214 leader_election.cc:288] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:55:30.551578 24218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:55:30.552924 24218 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [term 3 LEADER]: Becoming Leader. State: Replica: c3d47f285d274861a613789d7fbcf84f, State: Running, Role: LEADER
I20260926 20:55:30.553649 24218 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } }
I20260926 20:55:30.554003 24214 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:30.556265 24220 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3d47f285d274861a613789d7fbcf84f. Latest consensus state: current_term: 3 leader_uuid: "c3d47f285d274861a613789d7fbcf84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } } }
I20260926 20:55:30.556628 24220 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:30.557152 24219 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c3d47f285d274861a613789d7fbcf84f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3d47f285d274861a613789d7fbcf84f" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 40915 } } }
I20260926 20:55:30.557420 24219 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d47f285d274861a613789d7fbcf84f [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:30.571046 24224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:30.580832 24224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:30.582340 24224 catalog_manager.cc:1259] Loaded cluster ID: 3e4c427df9f34a34bb598376c69c56a1
I20260926 20:55:30.582439 24224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 20:55:30.584383 24224 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff1000989ad google::LogMessage::Fail() at ??:0
@ 0x7ff10009b88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff1000984de google::LogMessage::Flush() at ??:0
@ 0x7ff100099a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff115acf512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff115acc976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff115b54610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff115b54439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff102ce4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff102ce791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff102ce7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff102cc2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff105f9c6db start_thread at ??:0
@ 0x7ff0fef3261f clone at ??:0
W20260926 20:55:30.861645 24216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:30.861985 24216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:30.862155 24216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:30.869467 24216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:30.869620 24216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:30.869799 24216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:55:30.879565 24216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:33765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=37165
--tserver_master_addrs=127.8.249.190:40915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:30.880885 24216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:30.882182 24216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:30.895825 24256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:30.895808 24255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:30.895861 24258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:30.896390 24216 server_base.cc:1034] running on GCE node
I20260926 20:55:30.896991 24216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:30.897910 24216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:30.899107 24216 hybrid_clock.cc:648] HybridClock initialized: now 1790456130899066 us; error 32 us; skew 500 ppm
I20260926 20:55:30.899405 24216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:30 dist-test-slave-2dg3 krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456130, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:30.902271 24216 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:55:30.903690 24216 webserver.cc:460] Webserver started at http://127.8.249.129:37165/ using document root <none> and password file <none>
I20260926 20:55:30.904341 24216 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:30.904487 24216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:30.909693 24216 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 20:55:30.912444 24265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:30.913735 24216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:55:30.913900 24216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "a4fae9e9222c48cbaf2c6e34450ee724"
format_stamp: "Formatted at 2026-09-26 20:55:24 on dist-test-slave-2dg3"
server_key: "f44c777d5957e8ded95e50b7502ea58c"
server_key_iv: "ebb80d8037e3fd3d08d6e723499bf646"
server_key_version: "encryptionkey@0"
I20260926 20:55:30.914844 24216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:30.938845 24216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:30.943153 24216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:30.943423 24216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:30.944406 24216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:30.945751 24216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:30.945878 24216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:30.946015 24216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:30.946106 24216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:30.979549 24216 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:33765
I20260926 20:55:30.979636 24377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:33765 every 8 connection(s)
I20260926 20:55:30.981101 24216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:55:30.985895 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:30.983986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:56101)
0926 20:55:30.984784 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:30.985033 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":352,"threads_started":1}
W20260926 20:55:30.986563 24378 heartbeater.cc:644] Failed to heartbeat to 127.8.249.190:40915 (0 consecutive failures): Network error: Failed to ping master at 127.8.249.190:40915: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
I20260926 20:55:30.987422 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:30.987063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:42791)
0926 20:55:30.987258 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:30.987324 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 20:55:30.988224 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:30.987904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:35911)
0926 20:55:30.988046 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:30.988118 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 20:55:30.988526 24378 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:55:30.989007 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 24216
I20260926 20:55:30.989652 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:35155
--local_ip_for_outbound_sockets=127.8.249.130
--tserver_master_addrs=127.8.249.190:40915
--webserver_port=39071
--webserver_interface=127.8.249.130
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:31.315594 24381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:31.315883 24381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:31.315979 24381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:31.323345 24381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:31.323458 24381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:31.323571 24381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:55:31.333264 24381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:35155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=39071
--tserver_master_addrs=127.8.249.190:40915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:31.334555 24381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:31.335943 24381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:31.349095 24387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:31.349126 24386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:31.349166 24389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:31.350055 24381 server_base.cc:1034] running on GCE node
I20260926 20:55:31.350778 24381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:31.351931 24381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:31.353133 24381 hybrid_clock.cc:648] HybridClock initialized: now 1790456131353093 us; error 38 us; skew 500 ppm
I20260926 20:55:31.353464 24381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:31 dist-test-slave-2dg3 krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456131, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:31.356675 24381 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:55:31.358095 24381 webserver.cc:460] Webserver started at http://127.8.249.130:39071/ using document root <none> and password file <none>
I20260926 20:55:31.358852 24381 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:31.358997 24381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:31.364255 24381 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 20:55:31.367077 24396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:31.368379 24381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:55:31.368558 24381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "d992231bfaab4bf7a329e37633a4a30a"
format_stamp: "Formatted at 2026-09-26 20:55:25 on dist-test-slave-2dg3"
server_key: "ed62dfab1cce587383a682462666a6b1"
server_key_iv: "099fc7c9c2d6a0129e65c0bdee472198"
server_key_version: "encryptionkey@0"
I20260926 20:55:31.369529 24381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:31.394099 24381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:31.398506 24381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:31.398835 24381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:31.399838 24381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:31.401163 24381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:31.401305 24381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:31.401443 24381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:31.401543 24381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:31.434619 24381 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:35155
I20260926 20:55:31.434702 24508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:35155 every 8 connection(s)
I20260926 20:55:31.436120 24381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:55:31.440948 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:31.439151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:60575)
0926 20:55:31.439873 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:31.440143 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":278,"threads_started":1}
W20260926 20:55:31.441555 24509 heartbeater.cc:644] Failed to heartbeat to 127.8.249.190:40915 (0 consecutive failures): Network error: Failed to ping master at 127.8.249.190:40915: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
I20260926 20:55:31.442358 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:31.442029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:52279)
0926 20:55:31.442172 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:31.442241 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 20:55:31.443271 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:31.442934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:36167)
0926 20:55:31.443064 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:31.443140 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 20:55:31.443521 24509 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:55:31.443712 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 24381
I20260926 20:55:31.444263 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:39267
--local_ip_for_outbound_sockets=127.8.249.131
--tserver_master_addrs=127.8.249.190:40915
--webserver_port=38329
--webserver_interface=127.8.249.131
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:31.762580 24512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:31.762926 24512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:31.763132 24512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:31.770296 24512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:31.770419 24512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:31.770579 24512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:55:31.780299 24512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:39267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=38329
--tserver_master_addrs=127.8.249.190:40915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:31.781574 24512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:31.782966 24512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:31.796092 24517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:31.796025 24520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:31.796062 24518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:31.797199 24512 server_base.cc:1034] running on GCE node
I20260926 20:55:31.797706 24512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:31.798560 24512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:31.799787 24512 hybrid_clock.cc:648] HybridClock initialized: now 1790456131799708 us; error 69 us; skew 500 ppm
I20260926 20:55:31.800103 24512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:31 dist-test-slave-2dg3 krb5kdc[23069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456131, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:31.803067 24512 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:55:31.804500 24512 webserver.cc:460] Webserver started at http://127.8.249.131:38329/ using document root <none> and password file <none>
I20260926 20:55:31.805217 24512 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:31.805354 24512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:31.810534 24512 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:55:31.813066 24527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:31.814266 24512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 20:55:31.814440 24512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "7132b5eaf5ba4112ae5063e93d52b4e3"
format_stamp: "Formatted at 2026-09-26 20:55:25 on dist-test-slave-2dg3"
server_key: "51205398d2bee1c481f88c212f0a7c15"
server_key_iv: "89545a52e0880eeb1bd05eb9274d3b9a"
server_key_version: "encryptionkey@0"
I20260926 20:55:31.815511 24512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:31.842684 24512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:31.846629 24512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:31.846936 24512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:31.847954 24512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:31.849290 24512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:31.849409 24512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:31.849550 24512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:31.849637 24512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:31.882979 24512 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:39267
I20260926 20:55:31.883023 24639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:39267 every 8 connection(s)
I20260926 20:55:31.884413 24512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:55:31.886776 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 24512
I20260926 20:55:31.889652 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:31.887386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:51761)
0926 20:55:31.888317 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:31.888766 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":319,"threads_started":1}
W20260926 20:55:31.890350 24640 heartbeater.cc:644] Failed to heartbeat to 127.8.249.190:40915 (0 consecutive failures): Network error: Failed to ping master at 127.8.249.190:40915: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
I20260926 20:55:31.891192 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:31.890895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:40809)
0926 20:55:31.891029 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:31.891094 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:55:31.892005 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:31.891722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:36209)
0926 20:55:31.891828 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:31.891895 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:55:31.892253 24640 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:55:31.989928 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:31.989115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:50461)
0926 20:55:31.989587 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:31.989673 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":239,"threads_started":1}
I20260926 20:55:32.444741 24645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:32.444006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:42637)
0926 20:55:32.444409 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:32.444504 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":189,"threads_started":1}
I20260926 20:55:32.893556 24646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:32.892701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:57683)
0926 20:55:32.893210 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:32.893311 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":216,"threads_started":1}
I20260926 20:55:32.991664 24647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:32.990972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:46393)
0926 20:55:32.991423 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:32.991504 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":214,"threads_started":1}
I20260926 20:55:33.446246 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:33.445572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:47929)
0926 20:55:33.446009 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:33.446086 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":214,"threads_started":1}
I20260926 20:55:33.895334 24649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:33.894531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:46807)
0926 20:55:33.895096 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:33.895176 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":391,"threads_started":1}
I20260926 20:55:33.993271 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:33.992589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:51869)
0926 20:55:33.993014 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:33.993095 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":231,"threads_started":1}
I20260926 20:55:34.447867 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:34.447128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:38267)
0926 20:55:34.447590 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:34.447668 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":267,"threads_started":1}
I20260926 20:55:34.896898 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:34.896190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:44599)
0926 20:55:34.896637 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:34.896717 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":237,"threads_started":1}
I20260926 20:55:34.994993 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:34.994145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:42413)
0926 20:55:34.994655 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:34.994737 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":272,"threads_started":1}
I20260926 20:55:35.449522 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:35.448806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:54151)
0926 20:55:35.449249 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:35.449327 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":233,"threads_started":1}
I20260926 20:55:35.898728 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:35.897909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:40741)
0926 20:55:35.898380 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:35.898473 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":235,"threads_started":1}
I20260926 20:55:35.996819 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:35.995986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:36199)
0926 20:55:35.996523 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:35.996624 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":264,"threads_started":1}
I20260926 20:55:36.451264 24657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:36.450415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:59759)
0926 20:55:36.451014 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:36.451094 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":324,"threads_started":1}
I20260926 20:55:36.900466 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:36.899686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:46773)
0926 20:55:36.900212 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:36.900292 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":275,"threads_started":1}
I20260926 20:55:36.998541 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:36.997816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:37567)
0926 20:55:36.998270 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:36.998375 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
I20260926 20:55:37.453060 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:37.452331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:53979)
0926 20:55:37.452798 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:37.452890 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":226,"threads_started":1}
I20260926 20:55:37.902179 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:37.901438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:33133)
0926 20:55:37.901909 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:37.902005 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":252,"threads_started":1}
I20260926 20:55:38.000379 24662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:37.999522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:60839)
0926 20:55:38.000037 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:38.000126 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":242,"threads_started":1}
I20260926 20:55:38.454946 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:38.454025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:60593)
0926 20:55:38.454613 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:38.454698 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":253,"threads_started":1}
I20260926 20:55:38.903980 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:38.903203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:57905)
0926 20:55:38.903685 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:38.903775 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":203,"threads_started":1}
I20260926 20:55:39.002030 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:39.001306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:48501)
0926 20:55:39.001782 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:39.001868 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":253,"threads_started":1}
I20260926 20:55:39.456465 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:39.455800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:51725)
0926 20:55:39.456218 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:39.456295 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":215,"threads_started":1}
I20260926 20:55:39.906344 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:39.905213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:60331)
0926 20:55:39.905888 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:39.906052 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"spinlock_wait_cycles":31488,"thread_start_us":371,"threads_started":1}
I20260926 20:55:40.003890 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:40.003151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:54945)
0926 20:55:40.003625 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:40.003711 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":218,"threads_started":1}
I20260926 20:55:40.458046 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:40.457320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:51317)
0926 20:55:40.457796 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:40.457874 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":242,"threads_started":1}
I20260926 20:55:40.908154 24670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:40.907413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:56783)
0926 20:55:40.907884 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:40.907970 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":239,"threads_started":1}
I20260926 20:55:41.005590 24671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:41.004884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:43339)
0926 20:55:41.005342 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:41.005423 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":251,"threads_started":1}
I20260926 20:55:41.459692 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:41.459000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:51801)
0926 20:55:41.459419 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:41.459497 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":235,"threads_started":1}
I20260926 20:55:41.909937 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:41.909174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:55937)
0926 20:55:41.909683 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:41.909770 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":251,"threads_started":1}
I20260926 20:55:42.007236 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:42.006449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:43901)
0926 20:55:42.006962 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:42.007055 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":238,"threads_started":1}
I20260926 20:55:42.461568 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:42.460778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:45195)
0926 20:55:42.461277 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:42.461382 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":259,"threads_started":1}
I20260926 20:55:42.911773 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:42.910988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:55641)
0926 20:55:42.911495 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:42.911589 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":243,"threads_started":1}
I20260926 20:55:43.008881 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:43.008173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:34995)
0926 20:55:43.008627 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:43.008726 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":218,"threads_started":1}
I20260926 20:55:43.463346 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:43.462553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:52683)
0926 20:55:43.463102 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:43.463185 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":321,"threads_started":1}
I20260926 20:55:43.913673 24679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:43.912881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:54467)
0926 20:55:43.913366 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:43.913501 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":261,"threads_started":1}
I20260926 20:55:44.010565 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:44.009772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:36117)
0926 20:55:44.010274 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:44.010369 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":250,"threads_started":1}
I20260926 20:55:44.465396 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:44.464445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:54831)
0926 20:55:44.465044 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:44.465139 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":341,"threads_started":1}
I20260926 20:55:44.915377 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:44.914549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:57845)
0926 20:55:44.915098 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:44.915202 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":311,"threads_started":1}
I20260926 20:55:45.012369 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:45.011553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:52457)
0926 20:55:45.012090 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:45.012176 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":282,"threads_started":1}
I20260926 20:55:45.467211 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:45.466324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:55895)
0926 20:55:45.466895 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:45.467003 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":253,"threads_started":1}
I20260926 20:55:45.917203 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:45.916447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:51337)
0926 20:55:45.916898 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:45.917015 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":228,"threads_started":1}
I20260926 20:55:46.014345 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:46.013579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.129:50801)
0926 20:55:46.014035 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:46.014143 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":224,"threads_started":1}
I20260926 20:55:46.469301 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:46.468425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.130:56317)
0926 20:55:46.468913 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:46.469035 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":255,"threads_started":1}
I20260926 20:55:46.898447 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 24216
I20260926 20:55:46.931736 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:46.918616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:40915 (local address 127.8.249.131:40717)
0926 20:55:46.931382 (+ 12766us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:46.931520 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12613,"threads_started":1}
I20260926 20:55:46.932514 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:46.899671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.8.249.129:33765)
0926 20:55:46.900317 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:46.900330 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:46.900368 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:55:46.902064 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:46.902074 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:46.902504 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:46.902858 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:46.904236 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:46.905430 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:46.906930 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:46.907637 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:46.907778 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:46.921991 (+ 14213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:46.922006 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:46.922041 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:46.922121 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:46.927427 (+ 5306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:46.928624 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:46.928638 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:46.928653 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:46.928942 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:46.929441 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:46.929449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:46.929455 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:46.930413 (+ 958us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:46.930852 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:46.931330 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 20:55:46.932111 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":347,"threads_started":1}
I20260926 20:55:46.934162 24373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:47.034514 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 24216
I20260926 20:55:47.053874 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 24381
I20260926 20:55:47.072254 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:47.054964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.8.249.130:35155)
0926 20:55:47.055522 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:47.055536 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:47.055571 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:55:47.057185 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:47.057195 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:47.057627 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:47.057991 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:47.059596 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:47.060833 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:47.062141 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:47.062873 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:47.063073 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:47.065882 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:47.065890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:47.065905 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:47.065942 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:47.068796 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:47.069559 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:47.069569 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:47.069581 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:47.069748 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:47.070130 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:47.070137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:47.070140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:47.070653 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:47.070997 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:47.071530 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 20:55:47.071959 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":285,"threads_started":1}
I20260926 20:55:47.073807 24504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:47.158296 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 24381
I20260926 20:55:47.176069 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 24512
I20260926 20:55:47.194341 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:47.177217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.8.249.131:39267)
0926 20:55:47.177940 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:47.177954 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:47.178004 (+ 50us) server_negotiation.cc:407] Connection header received
0926 20:55:47.179679 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:47.179689 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:47.180137 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:47.180438 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:47.181849 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:47.183201 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:47.184498 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:47.185173 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:47.185310 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:47.188134 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:47.188141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:47.188157 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:47.188194 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:47.191208 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:47.191935 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:47.191942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:47.191949 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:47.192130 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:47.192456 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:47.192459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:47.192461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:47.192990 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:47.193234 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:47.193662 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 20:55:47.194030 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":380,"threads_started":1}
I20260926 20:55:47.196141 24635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:47.284703 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 24512
2026-09-26T20:55:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24669 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:55:47 dist-test-slave-2dg3 krb5kdc[24698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:55:47 dist-test-slave-2dg3 krb5kdc[24698](info): set up 2 sockets
Sep 26 20:55:47 dist-test-slave-2dg3 krb5kdc[24698](info): commencing operation
krb5kdc: starting...
W20260926 20:55:48.375411 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:55:48 dist-test-slave-2dg3 krb5kdc[24698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:55:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:55:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:55:48.620524 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:42571
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:42571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:48.947611 24711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:48.947916 24711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:48.948083 24711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:48.955405 24711 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:55:48.955518 24711 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:55:48.955556 24711 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:55:48.955662 24711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:55:48.955762 24711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:48.955855 24711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:48.955945 24711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:55:48.956041 24711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:55:48.966174 24711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:42571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:48.967510 24711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:48.968832 24711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:48.980312 24716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:48.980394 24719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:48.980363 24717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:48.981352 24711 server_base.cc:1034] running on GCE node
I20260926 20:55:48.982086 24711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:48.983457 24711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:48.984704 24711 hybrid_clock.cc:648] HybridClock initialized: now 1790456148984654 us; error 44 us; skew 500 ppm
I20260926 20:55:48.985023 24711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:48 dist-test-slave-2dg3 krb5kdc[24698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456148, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:48.988104 24711 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:55:48.989547 24711 webserver.cc:460] Webserver started at http://127.8.249.190:36095/ using document root <none> and password file <none>
I20260926 20:55:48.990211 24711 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:48.990350 24711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:48.990670 24711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:48.993541 24711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "8530959d1ef641089b77313224f7c483"
format_stamp: "Formatted at 2026-09-26 20:55:48 on dist-test-slave-2dg3"
server_key: "3a5f1cffc3d0cc27a309bf0796b09d70"
server_key_iv: "f7572c100d40d7c180639a42daca38cb"
server_key_version: "encryptionkey@0"
I20260926 20:55:48.994294 24711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "8530959d1ef641089b77313224f7c483"
format_stamp: "Formatted at 2026-09-26 20:55:48 on dist-test-slave-2dg3"
server_key: "3a5f1cffc3d0cc27a309bf0796b09d70"
server_key_iv: "f7572c100d40d7c180639a42daca38cb"
server_key_version: "encryptionkey@0"
I20260926 20:55:48.999194 24711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 20:55:49.002375 24726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:49.003693 24711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:55:49.003865 24711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "8530959d1ef641089b77313224f7c483"
format_stamp: "Formatted at 2026-09-26 20:55:48 on dist-test-slave-2dg3"
server_key: "3a5f1cffc3d0cc27a309bf0796b09d70"
server_key_iv: "f7572c100d40d7c180639a42daca38cb"
server_key_version: "encryptionkey@0"
I20260926 20:55:49.004076 24711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:49.016808 24711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:49.021291 24711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:49.021586 24711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:49.039500 24711 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:42571
I20260926 20:55:49.039477 24777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:42571 every 8 connection(s)
I20260926 20:55:49.040885 24711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:55:49.043896 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 24711
I20260926 20:55:49.044140 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:55:49.044802 9190 external_mini_cluster.cc:1428] Setting key 107536d5e9fae60d8923952dbc9ab75a
I20260926 20:55:49.044864 24778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:49.059612 24778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Bootstrap starting.
Sep 26 20:55:49 dist-test-slave-2dg3 krb5kdc[24698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:49.063651 24778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:49.065099 24778 log.cc:824] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:49.068809 24778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: No bootstrap required, opened a new log
I20260926 20:55:49.070554 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:49.046710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45720 (local address 127.8.249.190:42571)
0926 20:55:49.047561 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:49.047576 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:49.047627 (+ 51us) server_negotiation.cc:407] Connection header received
0926 20:55:49.049805 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:49.049834 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:49.050361 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:49.050840 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:49.052242 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:49.054220 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:49.055822 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:49.056972 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:49.057173 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:49.061507 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:49.061521 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:49.061544 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:49.061603 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:49.065664 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:49.066752 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:49.066762 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:49.066772 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:49.067022 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:49.067420 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:49.067424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:49.067428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:49.068131 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:49.068430 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:49.068740 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 20:55:49.069176 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":317,"threads_started":1}
I20260926 20:55:49.077383 24778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } }
I20260926 20:55:49.077728 24778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:49.077800 24778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8530959d1ef641089b77313224f7c483, State: Initialized, Role: FOLLOWER
I20260926 20:55:49.078405 24778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } }
I20260926 20:55:49.078616 24778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:55:49.078716 24778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:55:49.078913 24778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:49.081486 24778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } }
I20260926 20:55:49.081970 24778 leader_election.cc:302] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8530959d1ef641089b77313224f7c483; no voters:
I20260926 20:55:49.082617 24778 leader_election.cc:288] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:55:49.082770 24783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:49.083525 24783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 1 LEADER]: Becoming Leader. State: Replica: 8530959d1ef641089b77313224f7c483, State: Running, Role: LEADER
I20260926 20:55:49.084124 24783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } }
I20260926 20:55:49.085023 24778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:55:49.087502 24784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8530959d1ef641089b77313224f7c483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } } }
I20260926 20:55:49.087915 24784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:49.087464 24785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8530959d1ef641089b77313224f7c483. Latest consensus state: current_term: 1 leader_uuid: "8530959d1ef641089b77313224f7c483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } } }
I20260926 20:55:49.088979 24785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: This master's current role is: LEADER
I20260926 20:55:49.089839 24792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:55:49.094627 24792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:55:49.103657 24792 catalog_manager.cc:1347] Generated new cluster ID: 0c4a1dd3fdcb4fb7898db8f9743038c1
I20260926 20:55:49.103778 24792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:55:49.124286 24792 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:55:49.126289 24792 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:55:49.142386 24792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Generated new TSK 0
I20260926 20:55:49.143487 24792 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:55:49.202840 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:42571
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:49.539553 24804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:49.539902 24804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:49.540086 24804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:49.547350 24804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:49.547549 24804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:49.547745 24804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:55:49.558686 24804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:49.560071 24804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:49.561404 24804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:49.575141 24810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:49.575219 24809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:49.575773 24812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:49.576205 24804 server_base.cc:1034] running on GCE node
I20260926 20:55:49.576794 24804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:49.577708 24804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:49.578900 24804 hybrid_clock.cc:648] HybridClock initialized: now 1790456149578847 us; error 52 us; skew 500 ppm
I20260926 20:55:49.579180 24804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:49 dist-test-slave-2dg3 krb5kdc[24698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456149, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:49.582213 24804 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:55:49.583775 24804 webserver.cc:460] Webserver started at http://127.8.249.129:44111/ using document root <none> and password file <none>
I20260926 20:55:49.584501 24804 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:49.584645 24804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:49.584969 24804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:49.588145 24804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "523a940b1d2e484da650be3827da523f"
format_stamp: "Formatted at 2026-09-26 20:55:49 on dist-test-slave-2dg3"
server_key: "33259942dfb25255b015e0f26ab5f5c0"
server_key_iv: "e52953cbb66b6044cf74ed060ee3b792"
server_key_version: "encryptionkey@0"
I20260926 20:55:49.588935 24804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "523a940b1d2e484da650be3827da523f"
format_stamp: "Formatted at 2026-09-26 20:55:49 on dist-test-slave-2dg3"
server_key: "33259942dfb25255b015e0f26ab5f5c0"
server_key_iv: "e52953cbb66b6044cf74ed060ee3b792"
server_key_version: "encryptionkey@0"
I20260926 20:55:49.594084 24804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 20:55:49.597604 24819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:49.599119 24804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:55:49.599257 24804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "523a940b1d2e484da650be3827da523f"
format_stamp: "Formatted at 2026-09-26 20:55:49 on dist-test-slave-2dg3"
server_key: "33259942dfb25255b015e0f26ab5f5c0"
server_key_iv: "e52953cbb66b6044cf74ed060ee3b792"
server_key_version: "encryptionkey@0"
I20260926 20:55:49.599428 24804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:49.613297 24804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:49.617559 24804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:49.617812 24804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:49.618764 24804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:49.620280 24804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:49.620355 24804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:49.620440 24804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:49.620496 24804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:49.654692 24804 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:45715
I20260926 20:55:49.654745 24931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:45715 every 8 connection(s)
I20260926 20:55:49.656051 24804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:55:49.662904 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 24804
I20260926 20:55:49.663167 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:55:49.663971 9190 external_mini_cluster.cc:1428] Setting key 190fb368f598787f9a3fcad8409fdfea
Sep 26 20:55:49 dist-test-slave-2dg3 krb5kdc[24698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456149, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:49.677737 24934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:49.658658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:57393 (local address 127.8.249.190:42571)
0926 20:55:49.659193 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:49.659199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:49.660281 (+ 1082us) server_negotiation.cc:407] Connection header received
0926 20:55:49.662083 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:49.662088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:49.662165 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:49.662270 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:49.664777 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:49.665642 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:49.667510 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:49.667934 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:49.668082 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:49.671903 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:49.671908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:49.671911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:49.671946 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:49.674494 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:49.675352 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:49.675358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:49.675360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:49.675488 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:49.675937 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:49.675942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:49.675948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:49.676214 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:49.676371 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:49.677110 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 20:55:49.677359 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":255,"threads_started":1}
I20260926 20:55:49.678742 24935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:49.659321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:57393)
0926 20:55:49.660107 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:49.660155 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:49.661791 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:49.662693 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:49.662706 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:49.663590 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:49.664509 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:49.664537 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:49.665832 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:49.665840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:49.667343 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:49.667355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:49.668185 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:49.668194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:49.668441 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:49.669161 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:49.669185 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:49.671651 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:49.674656 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:49.674665 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:49.674684 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:49.675207 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:49.675604 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:49.675609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:49.675613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:49.675817 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:49.676345 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:49.676353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:49.676720 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 20:55:49.676995 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 20:55:49.677556 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":334,"threads_started":1}
I20260926 20:55:49.680651 24932 heartbeater.cc:347] Connected to a master server at 127.8.249.190:42571
I20260926 20:55:49.681039 24932 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:49.681908 24932 heartbeater.cc:511] Master 127.8.249.190:42571 requested a full tablet report, sending...
I20260926 20:55:49.684369 24743 ts_manager.cc:194] Registered new tserver with Master: 523a940b1d2e484da650be3827da523f (127.8.249.129:45715)
I20260926 20:55:49.686264 24743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:57393
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:55:49.718324 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:42571
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:50.059007 24938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:50.059280 24938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:50.059372 24938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:50.066532 24938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:50.066644 24938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:50.066754 24938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:55:50.076571 24938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:50.077860 24938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:50.079255 24938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:50.092865 24943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:50.092833 24944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:50.092923 24946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:50.093794 24938 server_base.cc:1034] running on GCE node
I20260926 20:55:50.094455 24938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:50.095496 24938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:50.096733 24938 hybrid_clock.cc:648] HybridClock initialized: now 1790456150096691 us; error 34 us; skew 500 ppm
I20260926 20:55:50.097087 24938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:50 dist-test-slave-2dg3 krb5kdc[24698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456150, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:50.100389 24938 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:55:50.101982 24938 webserver.cc:460] Webserver started at http://127.8.249.130:35937/ using document root <none> and password file <none>
I20260926 20:55:50.102662 24938 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:50.102923 24938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:50.103291 24938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:50.106410 24938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "7dcfc60c899f4b42b75bf9217d3580b1"
format_stamp: "Formatted at 2026-09-26 20:55:50 on dist-test-slave-2dg3"
server_key: "689894a0338cb0256f22032a92d89da2"
server_key_iv: "029e986112a53807d5c068e934d66bc8"
server_key_version: "encryptionkey@0"
I20260926 20:55:50.107261 24938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "7dcfc60c899f4b42b75bf9217d3580b1"
format_stamp: "Formatted at 2026-09-26 20:55:50 on dist-test-slave-2dg3"
server_key: "689894a0338cb0256f22032a92d89da2"
server_key_iv: "029e986112a53807d5c068e934d66bc8"
server_key_version: "encryptionkey@0"
I20260926 20:55:50.112701 24938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:55:50.116365 24953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:50.117717 24938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:55:50.117895 24938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "7dcfc60c899f4b42b75bf9217d3580b1"
format_stamp: "Formatted at 2026-09-26 20:55:50 on dist-test-slave-2dg3"
server_key: "689894a0338cb0256f22032a92d89da2"
server_key_iv: "029e986112a53807d5c068e934d66bc8"
server_key_version: "encryptionkey@0"
I20260926 20:55:50.118105 24938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:50.138545 24938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:50.142985 24938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:50.143293 24938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:50.144308 24938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:50.145677 24938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:50.145751 24938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:50.145905 24938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:50.146010 24938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:50.182837 24938 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:39631
I20260926 20:55:50.182852 25065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:39631 every 8 connection(s)
I20260926 20:55:50.184297 24938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:55:50.192803 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 24938
I20260926 20:55:50.193022 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:55:50.193462 9190 external_mini_cluster.cc:1428] Setting key 42b2be8a19a69a0f45082900b8f2b788
Sep 26 20:55:50 dist-test-slave-2dg3 krb5kdc[24698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456150, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:50.208284 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:50.187360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:45333 (local address 127.8.249.190:42571)
0926 20:55:50.187810 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:50.187815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:50.189186 (+ 1371us) server_negotiation.cc:407] Connection header received
0926 20:55:50.191299 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:50.191304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:50.191379 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:50.191483 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:50.193323 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.194108 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.196221 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.196996 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.197155 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:50.201139 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:50.201148 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:50.201155 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:50.201202 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:50.204572 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:50.205572 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:50.205576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:50.205579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:50.205707 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:50.206294 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:50.206298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:50.206300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:50.206601 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:50.206720 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:50.207709 (+ 989us) server_negotiation.cc:299] Negotiation successful
0926 20:55:50.207945 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":229,"threads_started":1}
I20260926 20:55:50.209698 25069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:50.187982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:45333)
0926 20:55:50.188998 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:50.189050 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:50.190979 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:50.191718 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:50.191727 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:50.192409 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:50.193117 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:50.193146 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.194277 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.194281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:50.196050 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:50.196063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.197272 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.197278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:50.197512 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:50.198284 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:50.198316 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:50.200893 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:50.204720 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:50.204730 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:50.204750 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:50.205423 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:50.205815 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:50.205823 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:50.205830 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:50.206088 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:50.206741 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:50.206752 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:50.207280 (+ 528us) client_negotiation.cc:769] Sending connection context
0926 20:55:50.207592 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 20:55:50.208221 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":302,"threads_started":1}
I20260926 20:55:50.211365 25066 heartbeater.cc:347] Connected to a master server at 127.8.249.190:42571
I20260926 20:55:50.211688 25066 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:50.212437 25066 heartbeater.cc:511] Master 127.8.249.190:42571 requested a full tablet report, sending...
I20260926 20:55:50.214082 24743 ts_manager.cc:194] Registered new tserver with Master: 7dcfc60c899f4b42b75bf9217d3580b1 (127.8.249.130:39631)
I20260926 20:55:50.215396 24743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:45333
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:55:50.250319 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:42571
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:55:50.585502 25072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:55:50.585887 25072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:55:50.586023 25072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:55:50.593652 25072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:55:50.593767 25072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:55:50.593884 25072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:55:50.603648 25072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:55:50.604919 25072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:55:50.606190 25072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:55:50.619518 25077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:50.619478 25078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:55:50.619493 25080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:55:50.620539 25072 server_base.cc:1034] running on GCE node
I20260926 20:55:50.621224 25072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:55:50.622334 25072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:55:50.623519 25072 hybrid_clock.cc:648] HybridClock initialized: now 1790456150623472 us; error 40 us; skew 500 ppm
I20260926 20:55:50.623857 25072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:55:50 dist-test-slave-2dg3 krb5kdc[24698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456150, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:55:50.627079 25072 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:55:50.628598 25072 webserver.cc:460] Webserver started at http://127.8.249.131:34841/ using document root <none> and password file <none>
I20260926 20:55:50.629258 25072 fs_manager.cc:359] Metadata directory not provided
I20260926 20:55:50.629359 25072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:55:50.629621 25072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:55:50.632619 25072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "fcb683d687b7430998ae80a23e1a46eb"
format_stamp: "Formatted at 2026-09-26 20:55:50 on dist-test-slave-2dg3"
server_key: "707f4fe479b114311c02e8ede4767d7d"
server_key_iv: "9ac46d7897e9d18f1c74e14673f15e92"
server_key_version: "encryptionkey@0"
I20260926 20:55:50.633332 25072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "fcb683d687b7430998ae80a23e1a46eb"
format_stamp: "Formatted at 2026-09-26 20:55:50 on dist-test-slave-2dg3"
server_key: "707f4fe479b114311c02e8ede4767d7d"
server_key_iv: "9ac46d7897e9d18f1c74e14673f15e92"
server_key_version: "encryptionkey@0"
I20260926 20:55:50.638618 25072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 20:55:50.642232 25087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:50.643925 25072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:55:50.644115 25072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "fcb683d687b7430998ae80a23e1a46eb"
format_stamp: "Formatted at 2026-09-26 20:55:50 on dist-test-slave-2dg3"
server_key: "707f4fe479b114311c02e8ede4767d7d"
server_key_iv: "9ac46d7897e9d18f1c74e14673f15e92"
server_key_version: "encryptionkey@0"
I20260926 20:55:50.644351 25072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:55:50.665767 25072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:55:50.670235 25072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:55:50.670572 25072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:55:50.671651 25072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:55:50.673026 25072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:55:50.673134 25072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:50.673285 25072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:55:50.673362 25072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:55:50.690116 24932 heartbeater.cc:503] Master 127.8.249.190:42571 was elected leader, sending a full tablet report...
I20260926 20:55:50.707669 25072 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:33033
I20260926 20:55:50.707692 25199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:33033 every 8 connection(s)
I20260926 20:55:50.709175 25072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:55:50.715462 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 25072
I20260926 20:55:50.715705 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:55:50.716166 9190 external_mini_cluster.cc:1428] Setting key 5a5565ce539b3e1b3628c2c7ce5c5757
Sep 26 20:55:50 dist-test-slave-2dg3 krb5kdc[24698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456150, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:55:50.730393 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:50.712174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:45553 (local address 127.8.249.190:42571)
0926 20:55:50.712644 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:50.712649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:50.713693 (+ 1044us) server_negotiation.cc:407] Connection header received
0926 20:55:50.715710 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:50.715716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:50.715818 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:50.715962 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:50.717578 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.718369 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.720094 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.720518 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.720663 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:50.724445 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:50.724450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:50.724454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:50.724488 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:50.727192 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:50.727993 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:50.728003 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:50.728006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:50.728130 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:50.728585 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:50.728590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:50.728598 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:50.728869 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:50.729041 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:50.729763 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 20:55:50.729997 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
I20260926 20:55:50.731475 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:50.712771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:45553)
0926 20:55:50.713514 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:50.713551 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:50.715391 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:50.716216 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:50.716225 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:50.716799 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:55:50.717391 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:50.717410 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.718563 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.718568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:50.719939 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:50.719952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.720777 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.720786 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:50.721057 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:50.721729 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:55:50.721759 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:55:50.724200 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:55:50.727349 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:50.727359 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:50.727378 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:50.727828 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:50.728250 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:55:50.728255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:55:50.728258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:55:50.728460 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:55:50.729008 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:55:50.729016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:55:50.729374 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 20:55:50.729652 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 20:55:50.730220 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":282,"threads_started":1}
I20260926 20:55:50.732981 25200 heartbeater.cc:347] Connected to a master server at 127.8.249.190:42571
I20260926 20:55:50.733294 25200 heartbeater.cc:464] Registering TS with master...
I20260926 20:55:50.734007 25200 heartbeater.cc:511] Master 127.8.249.190:42571 requested a full tablet report, sending...
I20260926 20:55:50.735466 24743 ts_manager.cc:194] Registered new tserver with Master: fcb683d687b7430998ae80a23e1a46eb (127.8.249.131:33033)
I20260926 20:55:50.736266 24743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:45553
I20260926 20:55:50.744664 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:55:50.762228 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:50.749452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.8.249.190:42571)
0926 20:55:50.749691 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:50.749697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:50.749969 (+ 272us) server_negotiation.cc:407] Connection header received
0926 20:55:50.750103 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:50.750107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:50.750172 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:50.750253 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:50.751759 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.752502 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.753842 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.754274 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.754370 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:50.756796 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:50.756801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:50.756804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:50.756840 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:50.759423 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:50.760214 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:50.760218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:50.760220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:50.760342 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:50.760749 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:50.760752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:50.760754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:50.761091 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:50.761281 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:50.761663 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 20:55:50.761845 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:55:50.766319 24743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:55:50.773164 24743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:55:50.799633 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:50.786337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.8.249.130:39631)
0926 20:55:50.790960 (+ 4623us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:50.790971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:50.791002 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:55:50.791118 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:50.791126 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:50.791364 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:50.791591 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:50.792453 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.794140 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.797123 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.798551 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.798685 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:50.798908 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:50.799152 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 20:55:50.799317 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4468,"thread_start_us":4317,"threads_started":1}
I20260926 20:55:50.799726 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:50.786126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:39631 (local address 127.0.0.1:39982)
0926 20:55:50.787735 (+ 1609us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:50.787793 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:50.788019 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:50.791635 (+ 3616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:50.791645 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:50.791680 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:50.792244 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:50.792260 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.794346 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.794350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:50.796947 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:50.796962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.798749 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.798756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:50.798961 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:50.798993 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 20:55:50.799107 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 20:55:50.799237 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1248,"mutex_wait_us":174,"thread_start_us":1058,"threads_started":1}
I20260926 20:55:50.799863 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:50.789607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.8.249.131:33033)
0926 20:55:50.791346 (+ 1739us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:50.791353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:50.791985 (+ 632us) server_negotiation.cc:407] Connection header received
0926 20:55:50.792220 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:50.792226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:50.792417 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:50.792613 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:50.793502 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.794783 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.796772 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.798189 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.798325 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:50.798590 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:50.799353 (+ 763us) server_negotiation.cc:299] Negotiation successful
0926 20:55:50.799521 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1586,"thread_start_us":1423,"threads_started":1}
I20260926 20:55:50.800171 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:50.786355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:45715 (local address 127.0.0.1:54014)
0926 20:55:50.788311 (+ 1956us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:50.788337 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:50.788481 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:50.791785 (+ 3304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:50.791791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:50.791818 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:50.792411 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:50.792422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.794211 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.794219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:50.797206 (+ 2987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:50.797219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.799317 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.799323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:50.799487 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:50.799519 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 20:55:50.799601 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:55:50.799746 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1797,"mutex_wait_us":103,"thread_start_us":1633,"threads_started":1}
I20260926 20:55:50.800460 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:50.789252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:33033 (local address 127.0.0.1:53584)
0926 20:55:50.791860 (+ 2608us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:50.791874 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:50.792045 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:50.792895 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:50.792899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:50.792924 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:50.793291 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:50.793302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.794983 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.794988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:50.796603 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:50.796615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.798381 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.798387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:50.798607 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:50.798663 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 20:55:50.798840 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 20:55:50.799045 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2463,"mutex_wait_us":5,"thread_start_us":2278,"threads_started":1}
I20260926 20:55:50.800791 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:50.787072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.8.249.129:45715)
0926 20:55:50.789578 (+ 2506us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:50.789587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:50.789615 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:55:50.789726 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:50.789735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:50.790013 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:50.790224 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:50.792619 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.794074 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.797386 (+ 3312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:50.798803 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:50.798948 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:50.799172 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:50.800272 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0926 20:55:50.800455 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2307,"thread_start_us":2877,"threads_started":1}
I20260926 20:55:50.803992 25001 tablet_service.cc:1467] Processing CreateTablet for tablet f99b17934f7c4e57a5fd6231b69b0f0c (DEFAULT_TABLE table=test-table [id=926528f590fd4e5bac85e1e2af8bec8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:50.804231 24867 tablet_service.cc:1467] Processing CreateTablet for tablet f99b17934f7c4e57a5fd6231b69b0f0c (DEFAULT_TABLE table=test-table [id=926528f590fd4e5bac85e1e2af8bec8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:50.804222 25135 tablet_service.cc:1467] Processing CreateTablet for tablet f99b17934f7c4e57a5fd6231b69b0f0c (DEFAULT_TABLE table=test-table [id=926528f590fd4e5bac85e1e2af8bec8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:55:50.805413 25001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99b17934f7c4e57a5fd6231b69b0f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:50.805599 25135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99b17934f7c4e57a5fd6231b69b0f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:50.805615 24867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99b17934f7c4e57a5fd6231b69b0f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:55:50.818590 25219 tablet_bootstrap.cc:492] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1: Bootstrap starting.
I20260926 20:55:50.822031 25219 tablet_bootstrap.cc:654] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:50.823544 25219 log.cc:824] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:50.824049 25221 tablet_bootstrap.cc:492] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb: Bootstrap starting.
I20260926 20:55:50.824280 25220 tablet_bootstrap.cc:492] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f: Bootstrap starting.
I20260926 20:55:50.827454 25219 tablet_bootstrap.cc:492] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1: No bootstrap required, opened a new log
I20260926 20:55:50.827759 25219 ts_tablet_manager.cc:1397] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 20:55:50.828267 25220 tablet_bootstrap.cc:654] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:50.828337 25221 tablet_bootstrap.cc:654] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb: Neither blocks nor log segments found. Creating new log.
I20260926 20:55:50.829627 25220 log.cc:824] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:50.829627 25221 log.cc:824] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb: Log is configured to *not* fsync() on all Append() calls
I20260926 20:55:50.832710 25221 tablet_bootstrap.cc:492] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb: No bootstrap required, opened a new log
I20260926 20:55:50.833042 25221 ts_tablet_manager.cc:1397] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260926 20:55:50.833097 25220 tablet_bootstrap.cc:492] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f: No bootstrap required, opened a new log
I20260926 20:55:50.833417 25220 ts_tablet_manager.cc:1397] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260926 20:55:50.838014 25219 raft_consensus.cc:348] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } }
I20260926 20:55:50.838531 25219 raft_consensus.cc:374] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:50.838665 25219 raft_consensus.cc:729] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dcfc60c899f4b42b75bf9217d3580b1, State: Initialized, Role: FOLLOWER
I20260926 20:55:50.839423 25219 consensus_queue.cc:260] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } }
I20260926 20:55:50.840651 25066 heartbeater.cc:503] Master 127.8.249.190:42571 was elected leader, sending a full tablet report...
I20260926 20:55:50.841348 25219 ts_tablet_manager.cc:1428] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 20:55:50.841822 25221 raft_consensus.cc:348] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } }
I20260926 20:55:50.842325 25221 raft_consensus.cc:374] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:50.842432 25221 raft_consensus.cc:729] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb683d687b7430998ae80a23e1a46eb, State: Initialized, Role: FOLLOWER
I20260926 20:55:50.843106 25221 consensus_queue.cc:260] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } }
I20260926 20:55:50.844290 25200 heartbeater.cc:503] Master 127.8.249.190:42571 was elected leader, sending a full tablet report...
I20260926 20:55:50.844236 25220 raft_consensus.cc:348] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } }
I20260926 20:55:50.844700 25220 raft_consensus.cc:374] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:55:50.844825 25220 raft_consensus.cc:729] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523a940b1d2e484da650be3827da523f, State: Initialized, Role: FOLLOWER
I20260926 20:55:50.845566 25220 consensus_queue.cc:260] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } }
I20260926 20:55:50.846143 25221 ts_tablet_manager.cc:1428] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260926 20:55:50.849035 25220 ts_tablet_manager.cc:1428] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f: Time spent starting tablet: real 0.015s user 0.004s sys 0.009s
W20260926 20:55:50.908531 24933 tablet.cc:2369] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:55:50.936739 25067 tablet.cc:2369] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:55:50.960959 25201 tablet.cc:2369] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:55:51.077535 25227 raft_consensus.cc:482] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:55:51.077877 25227 raft_consensus.cc:504] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } }
I20260926 20:55:51.079737 25227 leader_election.cc:288] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7dcfc60c899f4b42b75bf9217d3580b1 (127.8.249.130:39631), fcb683d687b7430998ae80a23e1a46eb (127.8.249.131:33033)
I20260926 20:55:51.089125 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:51.080112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:54525 (local address 127.8.249.130:39631)
0926 20:55:51.080332 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:51.080338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:51.081112 (+ 774us) server_negotiation.cc:407] Connection header received
0926 20:55:51.081445 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:51.081451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:51.081536 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:51.081601 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:51.082535 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.083650 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.086718 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.087972 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.088131 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:51.088205 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:51.088740 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 20:55:51.088872 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 20:55:51.089319 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:51.080385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:35527 (local address 127.8.249.131:33033)
0926 20:55:51.080633 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:51.080639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:51.081353 (+ 714us) server_negotiation.cc:407] Connection header received
0926 20:55:51.081577 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:51.081587 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:51.081649 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:51.081755 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:55:51.082459 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.083645 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.086717 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.087995 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.088133 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:51.088205 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:51.088954 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 20:55:51.089077 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 20:55:51.089454 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:51.079949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:39631 (local address 127.8.249.129:54525)
0926 20:55:51.080991 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:51.081016 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:51.081240 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:51.081792 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:51.081796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:51.081821 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:51.082412 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:51.082422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.083788 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.083792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:51.086534 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:51.086549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.088254 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.088258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:51.088502 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:51.088586 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 20:55:51.088686 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 20:55:51.088921 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":232,"thread_start_us":621,"threads_started":1}
I20260926 20:55:51.089453 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:51.080282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:33033 (local address 127.8.249.129:35527)
0926 20:55:51.081215 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0926 20:55:51.081234 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:55:51.081363 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:55:51.081773 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:55:51.081777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:55:51.081801 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:55:51.082251 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:51.082267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.083786 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.083792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:51.086538 (+ 2746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:55:51.086549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.088247 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.088254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:55:51.088498 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:51.088608 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 20:55:51.088686 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:55:51.088879 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":362,"thread_start_us":600,"threads_started":1}
I20260926 20:55:51.090634 25021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99b17934f7c4e57a5fd6231b69b0f0c" candidate_uuid: "523a940b1d2e484da650be3827da523f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" is_pre_election: true
I20260926 20:55:51.091084 25155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99b17934f7c4e57a5fd6231b69b0f0c" candidate_uuid: "523a940b1d2e484da650be3827da523f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb683d687b7430998ae80a23e1a46eb" is_pre_election: true
I20260926 20:55:51.091317 25021 raft_consensus.cc:2454] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 523a940b1d2e484da650be3827da523f in term 0.
I20260926 20:55:51.091629 25155 raft_consensus.cc:2454] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 523a940b1d2e484da650be3827da523f in term 0.
I20260926 20:55:51.092115 24823 leader_election.cc:302] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 523a940b1d2e484da650be3827da523f, 7dcfc60c899f4b42b75bf9217d3580b1; no voters:
I20260926 20:55:51.092550 25227 raft_consensus.cc:2781] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:55:51.092712 25227 raft_consensus.cc:482] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:55:51.092823 25227 raft_consensus.cc:3037] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:51.095050 25227 raft_consensus.cc:504] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } }
I20260926 20:55:51.095664 25227 leader_election.cc:288] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [CANDIDATE]: Term 1 election: Requested vote from peers 7dcfc60c899f4b42b75bf9217d3580b1 (127.8.249.130:39631), fcb683d687b7430998ae80a23e1a46eb (127.8.249.131:33033)
I20260926 20:55:51.096166 25155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99b17934f7c4e57a5fd6231b69b0f0c" candidate_uuid: "523a940b1d2e484da650be3827da523f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb683d687b7430998ae80a23e1a46eb"
I20260926 20:55:51.096166 25021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99b17934f7c4e57a5fd6231b69b0f0c" candidate_uuid: "523a940b1d2e484da650be3827da523f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dcfc60c899f4b42b75bf9217d3580b1"
I20260926 20:55:51.096379 25155 raft_consensus.cc:3037] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:51.096379 25021 raft_consensus.cc:3037] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:55:51.099573 25155 raft_consensus.cc:2454] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 523a940b1d2e484da650be3827da523f in term 1.
I20260926 20:55:51.099573 25021 raft_consensus.cc:2454] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 523a940b1d2e484da650be3827da523f in term 1.
I20260926 20:55:51.100014 24823 leader_election.cc:302] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 523a940b1d2e484da650be3827da523f, 7dcfc60c899f4b42b75bf9217d3580b1; no voters:
I20260926 20:55:51.100329 25227 raft_consensus.cc:2781] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:55:51.100798 25227 raft_consensus.cc:686] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [term 1 LEADER]: Becoming Leader. State: Replica: 523a940b1d2e484da650be3827da523f, State: Running, Role: LEADER
I20260926 20:55:51.101274 25227 consensus_queue.cc:237] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } }
I20260926 20:55:51.106164 24743 catalog_manager.cc:5495] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f reported cstate change: term changed from 0 to 1, leader changed from <none> to 523a940b1d2e484da650be3827da523f (127.8.249.129). New cstate: current_term: 1 leader_uuid: "523a940b1d2e484da650be3827da523f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } health_report { overall_health: UNKNOWN } } }
I20260926 20:55:51.165557 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:51.160013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.8.249.129:45715)
0926 20:55:51.160224 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:51.160229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:51.160247 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:55:51.160412 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:51.160416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:51.160515 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:51.160644 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:55:51.161037 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.161738 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.163213 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.163616 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.163715 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:51.164292 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:55:51.164828 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:55:51.164938 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:51.165067 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 20:55:51.165134 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 20:55:51.173938 25021 raft_consensus.cc:1261] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1 [term 1 FOLLOWER]: Refusing update from remote peer 523a940b1d2e484da650be3827da523f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:51.173934 25155 raft_consensus.cc:1261] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb [term 1 FOLLOWER]: Refusing update from remote peer 523a940b1d2e484da650be3827da523f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:55:51.175081 25230 consensus_queue.cc:1035] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:51.175433 25227 consensus_queue.cc:1035] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:55:51.206305 25234 mvcc.cc:205] Tried to move back new op lower bound from 7333708395197800448 to 7333708394914209792. Current Snapshot: MvccSnapshot[applied={T|T < 7333708395197800448}]
I20260926 20:55:51.217608 24743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:55:51.217909 24743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:55:51.221753 24743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Sending DeleteTablet for 3 replicas of tablet f99b17934f7c4e57a5fd6231b69b0f0c
I20260926 20:55:51.222985 25001 tablet_service.cc:1514] Processing DeleteTablet for tablet f99b17934f7c4e57a5fd6231b69b0f0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:51 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:39982
I20260926 20:55:51.223101 25135 tablet_service.cc:1514] Processing DeleteTablet for tablet f99b17934f7c4e57a5fd6231b69b0f0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:51 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:53584
I20260926 20:55:51.223250 24867 tablet_service.cc:1514] Processing DeleteTablet for tablet f99b17934f7c4e57a5fd6231b69b0f0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:55:51 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:54014
I20260926 20:55:51.223508 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 24804
I20260926 20:55:51.224584 25245 tablet_replica.cc:330] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f: stopping tablet replica
I20260926 20:55:51.225049 25245 raft_consensus.cc:2229] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:55:51.225826 25245 raft_consensus.cc:2258] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:51.227394 25243 tablet_replica.cc:330] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb: stopping tablet replica
I20260926 20:55:51.227829 25243 raft_consensus.cc:2229] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:55:51.228322 25243 raft_consensus.cc:2258] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:51.229030 25244 tablet_replica.cc:330] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1: stopping tablet replica
I20260926 20:55:51.229445 25244 raft_consensus.cc:2229] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:55:51.229918 25244 raft_consensus.cc:2258] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:55:51.231807 25245 ts_tablet_manager.cc:1905] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:51.232335 25243 ts_tablet_manager.cc:1905] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:55:51.239374 25243 ts_tablet_manager.cc:1918] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:51.239378 25245 ts_tablet_manager.cc:1918] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:51.239550 25245 log.cc:1196] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/f99b17934f7c4e57a5fd6231b69b0f0c
I20260926 20:55:51.239573 25244 ts_tablet_manager.cc:1905] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 20:55:51 dist-test-slave-2dg3 krb5kdc[24698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:55:51.240208 25245 ts_tablet_manager.cc:1939] T f99b17934f7c4e57a5fd6231b69b0f0c P 523a940b1d2e484da650be3827da523f: Deleting consensus metadata
I20260926 20:55:51.242219 24727 catalog_manager.cc:4841] TS 523a940b1d2e484da650be3827da523f (127.8.249.129:45715): tablet f99b17934f7c4e57a5fd6231b69b0f0c (table test-table [id=926528f590fd4e5bac85e1e2af8bec8c]) successfully deleted
I20260926 20:55:51.246451 25243 log.cc:1196] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/f99b17934f7c4e57a5fd6231b69b0f0c
I20260926 20:55:51.246542 25244 ts_tablet_manager.cc:1918] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:55:51.246773 25244 log.cc:1196] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/f99b17934f7c4e57a5fd6231b69b0f0c
I20260926 20:55:51.247148 25243 ts_tablet_manager.cc:1939] T f99b17934f7c4e57a5fd6231b69b0f0c P fcb683d687b7430998ae80a23e1a46eb: Deleting consensus metadata
I20260926 20:55:51.247447 25244 ts_tablet_manager.cc:1939] T f99b17934f7c4e57a5fd6231b69b0f0c P 7dcfc60c899f4b42b75bf9217d3580b1: Deleting consensus metadata
I20260926 20:55:51.249212 24728 catalog_manager.cc:4841] TS fcb683d687b7430998ae80a23e1a46eb (127.8.249.131:33033): tablet f99b17934f7c4e57a5fd6231b69b0f0c (table test-table [id=926528f590fd4e5bac85e1e2af8bec8c]) successfully deleted
I20260926 20:55:51.249436 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:51.224611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.8.249.129:45715)
0926 20:55:51.224826 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:51.224834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:51.226718 (+ 1884us) server_negotiation.cc:407] Connection header received
0926 20:55:51.226967 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:51.226975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:51.227077 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:51.227171 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:51.231346 (+ 4175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.232458 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.234448 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.235207 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.235297 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:51.241181 (+ 5884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:51.241194 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:51.241213 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:51.241263 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:51.245353 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:51.246353 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:51.246363 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:51.246366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:51.246569 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:51.247156 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:51.247163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:51.247166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:51.247695 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:51.247977 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:51.248747 (+ 770us) server_negotiation.cc:299] Negotiation successful
0926 20:55:51.249016 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:55:51.250072 24730 catalog_manager.cc:4841] TS 7dcfc60c899f4b42b75bf9217d3580b1 (127.8.249.130:39631): tablet f99b17934f7c4e57a5fd6231b69b0f0c (table test-table [id=926528f590fd4e5bac85e1e2af8bec8c]) successfully deleted
I20260926 20:55:51.250661 24927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:51.350222 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 24804
I20260926 20:55:51.365486 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 24938
Sep 26 20:55:51 dist-test-slave-2dg3 krb5kdc[24698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:55:51.381565 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:51.366344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.8.249.130:39631)
0926 20:55:51.366552 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:51.366558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:51.366658 (+ 100us) server_negotiation.cc:407] Connection header received
0926 20:55:51.366900 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:51.366904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:51.366969 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:51.367079 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:51.368426 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.369152 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.370522 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.371042 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.371177 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:51.375688 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:51.375704 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:51.375720 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:51.375761 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:51.378485 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:51.379380 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:51.379389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:51.379391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:51.379547 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:51.379980 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:51.379991 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:51.379994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:51.380349 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:51.380659 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:51.380985 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 20:55:51.381210 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 20:55:51.382236 25061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:51.471733 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 24938
I20260926 20:55:51.489352 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 25072
Sep 26 20:55:51 dist-test-slave-2dg3 krb5kdc[24698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:55:51.505246 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:55:51.490206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.8.249.131:33033)
0926 20:55:51.490438 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 20:55:51.490445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:55:51.490466 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:55:51.490659 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:55:51.490663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:55:51.490732 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:55:51.490883 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:55:51.492235 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.492929 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.494410 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:55:51.494882 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:55:51.495038 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:55:51.499775 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:55:51.499790 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:55:51.499807 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:55:51.499845 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:55:51.502340 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:51.503230 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:51.503239 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:51.503241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:51.503369 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:55:51.503745 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:55:51.503752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:55:51.503755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:55:51.504124 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:55:51.504439 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 20:55:51.504708 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 20:55:51.504917 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:55:51.505868 25195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:51.596123 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 25072
I20260926 20:55:51.612939 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 24711
I20260926 20:55:51.614192 24773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:55:51.725152 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 24711
I20260926 20:56:11.747148 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:42571
--webserver_interface=127.8.249.190
--webserver_port=36095
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:42571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:12.077440 25251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:12.077718 25251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:12.077805 25251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:12.085088 25251 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:56:12.085189 25251 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:56:12.085227 25251 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:56:12.085270 25251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:12.085326 25251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:12.085391 25251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:12.085434 25251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:12.085481 25251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:12.098543 25251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:42571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=36095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:12.099830 25251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:12.101052 25251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:12.124002 25258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:12.123963 25257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:12.124046 25260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:12.125113 25251 server_base.cc:1034] running on GCE node
I20260926 20:56:12.126076 25251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:12.127749 25251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:12.128999 25251 hybrid_clock.cc:648] HybridClock initialized: now 1790456172128931 us; error 45 us; skew 500 ppm
I20260926 20:56:12.129392 25251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:56:12 dist-test-slave-2dg3 krb5kdc[24698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456172, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:12.133014 25251 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:56:12.134738 25251 webserver.cc:460] Webserver started at http://127.8.249.190:36095/ using document root <none> and password file <none>
I20260926 20:56:12.135573 25251 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:12.135680 25251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:12.141870 25251 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 20:56:12.144950 25267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:12.146399 25251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:56:12.146592 25251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "8530959d1ef641089b77313224f7c483"
format_stamp: "Formatted at 2026-09-26 20:55:48 on dist-test-slave-2dg3"
server_key: "3a5f1cffc3d0cc27a309bf0796b09d70"
server_key_iv: "f7572c100d40d7c180639a42daca38cb"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.147795 25251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:12.187139 25251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:12.191807 25251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:12.192134 25251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:12.211316 25251 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:42571
I20260926 20:56:12.211404 25318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:42571 every 8 connection(s)
I20260926 20:56:12.212854 25251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:12.219658 25319 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:56:12.222121 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 25251
I20260926 20:56:12.223156 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:45715
--local_ip_for_outbound_sockets=127.8.249.129
--tserver_master_addrs=127.8.249.190:42571
--webserver_port=44111
--webserver_interface=127.8.249.129
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:56:12.223510 25319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Bootstrap starting.
I20260926 20:56:12.245280 25319 log.cc:824] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:12.257961 25319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:56:12.258551 25319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Bootstrap complete.
I20260926 20:56:12.267994 25319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } }
I20260926 20:56:12.268843 25319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8530959d1ef641089b77313224f7c483, State: Initialized, Role: FOLLOWER
I20260926 20:56:12.269515 25319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } }
I20260926 20:56:12.269761 25319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:12.269881 25319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:12.270090 25319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:56:12.273917 25319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } }
I20260926 20:56:12.274428 25319 leader_election.cc:302] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8530959d1ef641089b77313224f7c483; no voters:
I20260926 20:56:12.275134 25319 leader_election.cc:288] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:56:12.275264 25323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:56:12.276476 25323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 2 LEADER]: Becoming Leader. State: Replica: 8530959d1ef641089b77313224f7c483, State: Running, Role: LEADER
I20260926 20:56:12.277123 25323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } }
I20260926 20:56:12.277529 25319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:12.279795 25324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8530959d1ef641089b77313224f7c483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } } }
I20260926 20:56:12.280221 25324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:12.284098 25325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8530959d1ef641089b77313224f7c483. Latest consensus state: current_term: 2 leader_uuid: "8530959d1ef641089b77313224f7c483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } } }
I20260926 20:56:12.284598 25325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:12.286377 25329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:12.296252 25329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:12.297997 25329 catalog_manager.cc:1259] Loaded cluster ID: 0c4a1dd3fdcb4fb7898db8f9743038c1
I20260926 20:56:12.298213 25329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:12.303936 25329 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:12.305187 25329 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Loaded TSK: 0
I20260926 20:56:12.326332 25329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Generated new TSK 1
I20260926 20:56:12.326725 25329 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:56:12.584331 25321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:12.584602 25321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:12.584753 25321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:12.592445 25321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:12.592558 25321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:12.592726 25321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:56:12.602468 25321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:45715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=44111
--tserver_master_addrs=127.8.249.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:12.603820 25321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:12.605144 25321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:12.619617 25346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:12.619645 25347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:12.620335 25349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:12.620559 25321 server_base.cc:1034] running on GCE node
I20260926 20:56:12.621217 25321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:12.622176 25321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:12.623399 25321 hybrid_clock.cc:648] HybridClock initialized: now 1790456172623336 us; error 58 us; skew 500 ppm
I20260926 20:56:12.623706 25321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:56:12 dist-test-slave-2dg3 krb5kdc[24698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456172, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:12.626745 25321 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:56:12.628223 25321 webserver.cc:460] Webserver started at http://127.8.249.129:44111/ using document root <none> and password file <none>
I20260926 20:56:12.628867 25321 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:12.629024 25321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:12.634114 25321 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 20:56:12.637082 25356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:12.638438 25321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:56:12.638624 25321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "523a940b1d2e484da650be3827da523f"
format_stamp: "Formatted at 2026-09-26 20:55:49 on dist-test-slave-2dg3"
server_key: "33259942dfb25255b015e0f26ab5f5c0"
server_key_iv: "e52953cbb66b6044cf74ed060ee3b792"
server_key_version: "encryptionkey@0"
I20260926 20:56:12.639691 25321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:12.665282 25321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:12.669317 25321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:12.669617 25321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:12.670564 25321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:12.671933 25321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:12.672026 25321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:12.672107 25321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:12.672161 25321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:12.706003 25321 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:45715
I20260926 20:56:12.706058 25468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:45715 every 8 connection(s)
I20260926 20:56:12.707602 25321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:12.710649 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 25321
I20260926 20:56:12.711553 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:39631
--local_ip_for_outbound_sockets=127.8.249.130
--tserver_master_addrs=127.8.249.190:42571
--webserver_port=35937
--webserver_interface=127.8.249.130
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:56:12 dist-test-slave-2dg3 krb5kdc[24698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456172, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:56:12.734335 25472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:12.710829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:37509)
0926 20:56:12.712349 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:12.712405 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:12.715241 (+ 2836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:12.716772 (+ 1531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:12.716787 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:12.717593 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:12.718397 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:12.718424 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.720232 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.720237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:12.721481 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:12.721505 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.722624 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.722632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:12.722927 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:12.723652 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:12.723675 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:12.726186 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:12.729849 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:12.729860 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:12.729877 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:12.730332 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:12.730812 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:12.730819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:12.730823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:12.731022 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:12.731771 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:12.731779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:12.732121 (+ 342us) client_negotiation.cc:769] Sending connection context
0926 20:56:12.732382 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 20:56:12.732965 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1187,"thread_start_us":953,"threads_started":1}
I20260926 20:56:12.734335 25471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:12.710679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:37509 (local address 127.8.249.190:42571)
0926 20:56:12.711395 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:12.711413 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:12.712571 (+ 1158us) server_negotiation.cc:407] Connection header received
0926 20:56:12.715856 (+ 3285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:12.715871 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:12.716236 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:12.716539 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:12.718615 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.720057 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.721673 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:12.722362 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:12.722571 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:12.726496 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:12.726504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:12.726524 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:12.726563 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:12.729664 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:12.730488 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:12.730496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:12.730502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:12.730674 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:12.731142 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:12.731148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:12.731151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:12.731625 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:12.731839 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:12.732607 (+ 768us) server_negotiation.cc:299] Negotiation successful
0926 20:56:12.732980 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":296,"threads_started":1}
I20260926 20:56:12.737309 25469 heartbeater.cc:347] Connected to a master server at 127.8.249.190:42571
I20260926 20:56:12.737665 25469 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:12.738528 25469 heartbeater.cc:511] Master 127.8.249.190:42571 requested a full tablet report, sending...
I20260926 20:56:12.740942 25284 ts_manager.cc:194] Registered new tserver with Master: 523a940b1d2e484da650be3827da523f (127.8.249.129:45715)
I20260926 20:56:12.742456 25284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:37509
W20260926 20:56:13.042279 25473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:13.042680 25473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:13.042865 25473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:13.050493 25473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:13.050607 25473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:13.050721 25473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:56:13.060705 25473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:39631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=35937
--tserver_master_addrs=127.8.249.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:13.061889 25473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:13.063134 25473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:13.075697 25478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:13.075775 25481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:13.075742 25479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:13.076409 25473 server_base.cc:1034] running on GCE node
I20260926 20:56:13.077128 25473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:13.078236 25473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:13.079456 25473 hybrid_clock.cc:648] HybridClock initialized: now 1790456173079412 us; error 59 us; skew 500 ppm
I20260926 20:56:13.079783 25473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:56:13 dist-test-slave-2dg3 krb5kdc[24698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456173, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:13.082902 25473 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:56:13.084419 25473 webserver.cc:460] Webserver started at http://127.8.249.130:35937/ using document root <none> and password file <none>
I20260926 20:56:13.085099 25473 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:13.085249 25473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:13.090937 25473 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:56:13.093654 25488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.094944 25473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:13.095108 25473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "7dcfc60c899f4b42b75bf9217d3580b1"
format_stamp: "Formatted at 2026-09-26 20:55:50 on dist-test-slave-2dg3"
server_key: "689894a0338cb0256f22032a92d89da2"
server_key_iv: "029e986112a53807d5c068e934d66bc8"
server_key_version: "encryptionkey@0"
I20260926 20:56:13.096069 25473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:13.115351 25473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:13.119470 25473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:13.119755 25473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:13.120734 25473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:13.122066 25473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:13.122201 25473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.122334 25473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:13.122424 25473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.155352 25473 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:39631
I20260926 20:56:13.155411 25600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:39631 every 8 connection(s)
I20260926 20:56:13.156839 25473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:13.158375 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 25473
I20260926 20:56:13.159126 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:33033
--local_ip_for_outbound_sockets=127.8.249.131
--tserver_master_addrs=127.8.249.190:42571
--webserver_port=34841
--webserver_interface=127.8.249.131
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:56:13 dist-test-slave-2dg3 krb5kdc[24698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456173, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:56:13.180197 25471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.160514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:59451 (local address 127.8.249.190:42571)
0926 20:56:13.160715 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.160720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.161026 (+ 306us) server_negotiation.cc:407] Connection header received
0926 20:56:13.163943 (+ 2917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.163948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.164032 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.164124 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:13.166285 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.167075 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.168912 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.169352 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.169459 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.173998 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:13.174003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:13.174006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:13.174061 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:13.176726 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:13.177670 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:13.177677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:13.177681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:13.177850 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:13.178316 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:13.178321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:13.178324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:13.178690 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:13.178878 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.179587 (+ 709us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.179829 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 20:56:13.181275 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.160015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:59451)
0926 20:56:13.160841 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.160898 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:13.163586 (+ 2688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:13.164431 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:13.164446 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:13.165256 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:13.166082 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.166109 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.167235 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.167244 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.168737 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.168754 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.169606 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.169616 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.169962 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.170772 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:13.170838 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:13.173687 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:13.176916 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:13.176926 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:13.176947 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:13.177545 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:13.177965 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:13.177970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:13.177974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:13.178186 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:13.178849 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:13.178857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:13.179223 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 20:56:13.179503 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 20:56:13.180080 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":309,"threads_started":1}
I20260926 20:56:13.182866 25601 heartbeater.cc:347] Connected to a master server at 127.8.249.190:42571
I20260926 20:56:13.183197 25601 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:13.184062 25601 heartbeater.cc:511] Master 127.8.249.190:42571 requested a full tablet report, sending...
I20260926 20:56:13.185606 25284 ts_manager.cc:194] Registered new tserver with Master: 7dcfc60c899f4b42b75bf9217d3580b1 (127.8.249.130:39631)
I20260926 20:56:13.186861 25284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:59451
W20260926 20:56:13.495245 25604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:13.495515 25604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:13.495605 25604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:13.502599 25604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:13.502707 25604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:13.502839 25604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:56:13.512487 25604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:33033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=34841
--tserver_master_addrs=127.8.249.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:13.514170 25604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:13.515542 25604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:13.528640 25612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:13.528595 25610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:13.528682 25609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:13.529455 25604 server_base.cc:1034] running on GCE node
I20260926 20:56:13.530009 25604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:13.530975 25604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:13.532204 25604 hybrid_clock.cc:648] HybridClock initialized: now 1790456173532155 us; error 42 us; skew 500 ppm
I20260926 20:56:13.532567 25604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:56:13 dist-test-slave-2dg3 krb5kdc[24698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456173, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:13.541904 25604 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:56:13.543722 25604 webserver.cc:460] Webserver started at http://127.8.249.131:34841/ using document root <none> and password file <none>
I20260926 20:56:13.544534 25604 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:13.544697 25604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:13.550227 25604 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 20:56:13.553272 25619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.554709 25604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:56:13.554939 25604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "fcb683d687b7430998ae80a23e1a46eb"
format_stamp: "Formatted at 2026-09-26 20:55:50 on dist-test-slave-2dg3"
server_key: "707f4fe479b114311c02e8ede4767d7d"
server_key_iv: "9ac46d7897e9d18f1c74e14673f15e92"
server_key_version: "encryptionkey@0"
I20260926 20:56:13.556046 25604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:13.572667 25604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:13.577076 25604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:13.577409 25604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:13.578383 25604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:13.579751 25604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:13.579859 25604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.579986 25604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:13.580080 25604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:13.612226 25604 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:33033
I20260926 20:56:13.612267 25731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:33033 every 8 connection(s)
I20260926 20:56:13.613770 25604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:13.616907 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 25604
Sep 26 20:56:13 dist-test-slave-2dg3 krb5kdc[24698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456173, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:56:13.638545 25471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.616459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:37183 (local address 127.8.249.190:42571)
0926 20:56:13.616694 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.616702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.618143 (+ 1441us) server_negotiation.cc:407] Connection header received
0926 20:56:13.620254 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.620260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.620363 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.620485 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:13.622264 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.623521 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.626195 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.626989 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.627159 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.631496 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:13.631503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:13.631508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:13.631555 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:13.634992 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:13.636004 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:13.636013 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:13.636015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:13.636159 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:13.636706 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:13.636709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:13.636711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:13.637002 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:13.637173 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.637992 (+ 819us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.638233 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 20:56:13.639964 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.617075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:37183)
0926 20:56:13.617961 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.618026 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:13.619954 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:13.620906 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:13.620915 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:13.621450 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:13.622063 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.622082 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.623749 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.623756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.626040 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.626053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.627263 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.627274 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.627621 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.628393 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:13.628424 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:13.631264 (+ 2840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:13.635139 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:13.635151 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:13.635172 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:13.635844 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:13.636300 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:13.636305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:13.636309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:13.636587 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:13.637184 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:13.637191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:13.637528 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 20:56:13.637875 (+ 347us) client_negotiation.cc:240] Negotiation successful
0926 20:56:13.638391 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":362,"threads_started":1}
I20260926 20:56:13.640015 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.617800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48172 (local address 127.8.249.190:42571)
0926 20:56:13.620669 (+ 2869us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.620675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.623322 (+ 2647us) server_negotiation.cc:407] Connection header received
0926 20:56:13.623478 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.623486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.623567 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.623670 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:13.625868 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.627084 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.629263 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.629765 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.629937 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.633477 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:13.633485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:13.633492 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:13.633549 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:13.636983 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:13.637679 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:13.637685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:13.637689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:13.637837 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:13.638285 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:13.638294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:13.638300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:13.638767 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:13.639081 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.639366 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.639596 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2752,"spinlock_wait_cycles":2304,"thread_start_us":2633,"threads_started":1}
I20260926 20:56:13.641439 25732 heartbeater.cc:347] Connected to a master server at 127.8.249.190:42571
I20260926 20:56:13.641820 25732 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:13.642647 25732 heartbeater.cc:511] Master 127.8.249.190:42571 requested a full tablet report, sending...
I20260926 20:56:13.644377 25283 ts_manager.cc:194] Registered new tserver with Master: fcb683d687b7430998ae80a23e1a46eb (127.8.249.131:33033)
I20260926 20:56:13.645242 25283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:37183
I20260926 20:56:13.655783 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:56:13.662410 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.656782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.8.249.190:42571)
0926 20:56:13.657024 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.657031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.657320 (+ 289us) server_negotiation.cc:407] Connection header received
0926 20:56:13.657508 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.657511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.657658 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.657788 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:13.658196 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.659082 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.660656 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.661092 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.661185 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.661629 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:13.661826 (+ 197us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 20:56:13.662108 (+ 282us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48188: token expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 20:56:13.662561 25735 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48188: token expired
W20260926 20:56:13.663064 25207 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.249.190:42571)
I20260926 20:56:13.675489 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.663556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.8.249.190:42571)
0926 20:56:13.663708 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.663712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.663739 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:56:13.663775 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.663778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.663829 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.663903 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:13.665286 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.665942 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.667385 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.667742 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.667827 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.670351 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:13.670356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:13.670359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:13.670389 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:13.672960 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:13.673604 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:13.673608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:13.673610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:13.673730 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:13.674061 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:13.674064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:13.674066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:13.674354 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:13.674542 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.674960 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.675129 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:56:13.678771 25283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:13.686056 25283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:13.711035 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.697742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:33033 (local address 127.0.0.1:45980)
0926 20:56:13.699574 (+ 1832us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.699624 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:13.699902 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:13.702440 (+ 2538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:13.702451 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:13.702494 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:13.703110 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.703128 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.705609 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.705614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.707989 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.708011 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.710086 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.710093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.710303 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.710362 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 20:56:13.710481 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 20:56:13.710672 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1500,"thread_start_us":1326,"threads_started":1}
I20260926 20:56:13.711943 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.698079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.8.249.131:33033)
0926 20:56:13.700665 (+ 2586us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.700678 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.700719 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:56:13.701413 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.701431 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.701774 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.702051 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:13.703391 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.705417 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.708178 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.709589 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.709711 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.709937 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.711440 (+ 1503us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.711618 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2405,"thread_start_us":2241,"threads_started":1}
I20260926 20:56:13.718027 25667 tablet_service.cc:1467] Processing CreateTablet for tablet 2c51fb4243e34246ae4d37b8943845b5 (DEFAULT_TABLE table=test-table [id=834e306b7bce4bdd90f2bf250473aea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:13.719609 25667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c51fb4243e34246ae4d37b8943845b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:13.723795 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.697944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:45715 (local address 127.0.0.1:40852)
0926 20:56:13.712607 (+ 14663us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.712628 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:13.712845 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:13.715367 (+ 2522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:13.715374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:13.715413 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:13.715658 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.715672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.718629 (+ 2957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.718634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.720649 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.720663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.722883 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.722890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.723085 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.723126 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 20:56:13.723232 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 20:56:13.723390 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14545,"thread_start_us":19976,"threads_started":1}
I20260926 20:56:13.724282 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.698516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.8.249.129:45715)
0926 20:56:13.700175 (+ 1659us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.700183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.712722 (+ 12539us) server_negotiation.cc:407] Connection header received
0926 20:56:13.714327 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.714340 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.714664 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.714879 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:13.716650 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.718421 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.720828 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.722346 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.722505 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.722753 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.723413 (+ 660us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.723742 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1495,"thread_start_us":1358,"threads_started":1}
I20260926 20:56:13.727440 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.701057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.8.249.130:39631)
0926 20:56:13.718494 (+ 17437us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:13.718505 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:13.718536 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:56:13.718634 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:13.718642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:13.718942 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:13.719152 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:13.719953 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.721399 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.723908 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.726304 (+ 2396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.726471 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.726671 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:13.726985 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 20:56:13.727158 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17275,"thread_start_us":17178,"threads_started":1}
I20260926 20:56:13.727430 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:13.700886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:39631 (local address 127.0.0.1:44170)
0926 20:56:13.716308 (+ 15422us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:13.716332 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:13.716461 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:13.719391 (+ 2930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:13.719397 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:13.719424 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:13.719767 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.719778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.721573 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.721577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.723606 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:13.723624 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:13.726541 (+ 2917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:13.726547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:13.726743 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:13.726776 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 20:56:13.726929 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 20:56:13.727077 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15247,"thread_start_us":19906,"threads_started":1}
I20260926 20:56:13.728680 25404 tablet_service.cc:1467] Processing CreateTablet for tablet 2c51fb4243e34246ae4d37b8943845b5 (DEFAULT_TABLE table=test-table [id=834e306b7bce4bdd90f2bf250473aea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:13.730389 25404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c51fb4243e34246ae4d37b8943845b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:13.731145 25536 tablet_service.cc:1467] Processing CreateTablet for tablet 2c51fb4243e34246ae4d37b8943845b5 (DEFAULT_TABLE table=test-table [id=834e306b7bce4bdd90f2bf250473aea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:13.732244 25536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c51fb4243e34246ae4d37b8943845b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:13.737867 25748 tablet_bootstrap.cc:492] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb: Bootstrap starting.
I20260926 20:56:13.741634 25748 tablet_bootstrap.cc:654] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:13.742985 25748 log.cc:824] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:13.745513 25750 tablet_bootstrap.cc:492] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: Bootstrap starting.
I20260926 20:56:13.745846 25469 heartbeater.cc:503] Master 127.8.249.190:42571 was elected leader, sending a full tablet report...
I20260926 20:56:13.746889 25748 tablet_bootstrap.cc:492] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb: No bootstrap required, opened a new log
I20260926 20:56:13.747375 25748 ts_tablet_manager.cc:1397] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 20:56:13.751888 25751 tablet_bootstrap.cc:492] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1: Bootstrap starting.
I20260926 20:56:13.752398 25750 tablet_bootstrap.cc:654] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:13.753633 25750 log.cc:824] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:13.756150 25751 tablet_bootstrap.cc:654] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:13.756369 25750 tablet_bootstrap.cc:492] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: No bootstrap required, opened a new log
I20260926 20:56:13.756644 25750 ts_tablet_manager.cc:1397] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 20:56:13.757328 25751 log.cc:824] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:13.758924 25748 raft_consensus.cc:348] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } }
I20260926 20:56:13.759361 25748 raft_consensus.cc:374] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:13.759469 25748 raft_consensus.cc:729] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb683d687b7430998ae80a23e1a46eb, State: Initialized, Role: FOLLOWER
I20260926 20:56:13.760093 25748 consensus_queue.cc:260] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } }
I20260926 20:56:13.760666 25751 tablet_bootstrap.cc:492] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1: No bootstrap required, opened a new log
I20260926 20:56:13.761000 25751 ts_tablet_manager.cc:1397] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260926 20:56:13.765038 25750 raft_consensus.cc:348] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } }
I20260926 20:56:13.765388 25750 raft_consensus.cc:374] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:13.765548 25750 raft_consensus.cc:729] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523a940b1d2e484da650be3827da523f, State: Initialized, Role: FOLLOWER
I20260926 20:56:13.766106 25732 heartbeater.cc:503] Master 127.8.249.190:42571 was elected leader, sending a full tablet report...
I20260926 20:56:13.766294 25750 consensus_queue.cc:260] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } }
I20260926 20:56:13.766708 25748 ts_tablet_manager.cc:1428] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260926 20:56:13.769260 25750 ts_tablet_manager.cc:1428] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260926 20:56:13.772357 25751 raft_consensus.cc:348] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } }
I20260926 20:56:13.772748 25751 raft_consensus.cc:374] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:13.772866 25751 raft_consensus.cc:729] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dcfc60c899f4b42b75bf9217d3580b1, State: Initialized, Role: FOLLOWER
I20260926 20:56:13.773403 25751 consensus_queue.cc:260] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } }
I20260926 20:56:13.774379 25601 heartbeater.cc:503] Master 127.8.249.190:42571 was elected leader, sending a full tablet report...
I20260926 20:56:13.775094 25751 ts_tablet_manager.cc:1428] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
W20260926 20:56:13.865518 25733 tablet.cc:2369] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:56:13.908962 25602 tablet.cc:2369] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:56:13.960035 25470 tablet.cc:2369] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:14.005765 25755 raft_consensus.cc:482] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:14.006073 25755 raft_consensus.cc:504] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } }
I20260926 20:56:14.007845 25755 leader_election.cc:288] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcb683d687b7430998ae80a23e1a46eb (127.8.249.131:33033), 7dcfc60c899f4b42b75bf9217d3580b1 (127.8.249.130:39631)
I20260926 20:56:14.015409 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.008406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:47777 (local address 127.8.249.130:39631)
0926 20:56:14.008620 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.008627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.008992 (+ 365us) server_negotiation.cc:407] Connection header received
0926 20:56:14.009277 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.009284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.009371 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.009483 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:14.010128 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.011068 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.013229 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.014468 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.014633 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.014715 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.015044 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.015165 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:56:14.015707 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.008408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:40263 (local address 127.8.249.131:33033)
0926 20:56:14.008620 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.008627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.009634 (+ 1007us) server_negotiation.cc:407] Connection header received
0926 20:56:14.009799 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.009804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.009889 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.009996 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:14.010592 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.011442 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.013538 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.014761 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.014980 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.015055 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.015333 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.015472 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 20:56:14.016033 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.008169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:33033 (local address 127.8.249.129:40263)
0926 20:56:14.009507 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.009525 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:14.009659 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:14.010012 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:14.010015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:14.010064 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:14.010434 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:14.010445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.011586 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.011589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:14.013278 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:14.013290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.015006 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.015013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:14.015175 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.015203 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 20:56:14.015293 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:56:14.015439 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"mutex_wait_us":34,"thread_start_us":924,"threads_started":1}
I20260926 20:56:14.016135 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.008169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:39631 (local address 127.8.249.129:47777)
0926 20:56:14.008844 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.008871 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:14.009076 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:14.009528 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:14.009532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:14.009555 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:14.009975 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:14.009989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.011207 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.011212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:14.013077 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:14.013088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.014666 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.014671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:14.014849 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.014894 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:56:14.015004 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 20:56:14.015164 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":1,"thread_start_us":344,"threads_started":1}
I20260926 20:56:14.017212 25556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c51fb4243e34246ae4d37b8943845b5" candidate_uuid: "523a940b1d2e484da650be3827da523f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" is_pre_election: true
I20260926 20:56:14.017211 25687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c51fb4243e34246ae4d37b8943845b5" candidate_uuid: "523a940b1d2e484da650be3827da523f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb683d687b7430998ae80a23e1a46eb" is_pre_election: true
I20260926 20:56:14.017798 25556 raft_consensus.cc:2454] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 523a940b1d2e484da650be3827da523f in term 0.
I20260926 20:56:14.017798 25687 raft_consensus.cc:2454] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 523a940b1d2e484da650be3827da523f in term 0.
I20260926 20:56:14.018486 25358 leader_election.cc:302] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 523a940b1d2e484da650be3827da523f, fcb683d687b7430998ae80a23e1a46eb; no voters:
I20260926 20:56:14.018937 25755 raft_consensus.cc:2781] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:14.019129 25755 raft_consensus.cc:482] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:14.019233 25755 raft_consensus.cc:3037] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:14.021514 25755 raft_consensus.cc:504] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } }
I20260926 20:56:14.022171 25755 leader_election.cc:288] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [CANDIDATE]: Term 1 election: Requested vote from peers fcb683d687b7430998ae80a23e1a46eb (127.8.249.131:33033), 7dcfc60c899f4b42b75bf9217d3580b1 (127.8.249.130:39631)
I20260926 20:56:14.022687 25687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c51fb4243e34246ae4d37b8943845b5" candidate_uuid: "523a940b1d2e484da650be3827da523f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcb683d687b7430998ae80a23e1a46eb"
I20260926 20:56:14.022688 25556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c51fb4243e34246ae4d37b8943845b5" candidate_uuid: "523a940b1d2e484da650be3827da523f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dcfc60c899f4b42b75bf9217d3580b1"
I20260926 20:56:14.022928 25687 raft_consensus.cc:3037] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:14.022933 25556 raft_consensus.cc:3037] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:14.026180 25687 raft_consensus.cc:2454] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 523a940b1d2e484da650be3827da523f in term 1.
I20260926 20:56:14.026180 25556 raft_consensus.cc:2454] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 523a940b1d2e484da650be3827da523f in term 1.
I20260926 20:56:14.026633 25360 leader_election.cc:302] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 523a940b1d2e484da650be3827da523f, 7dcfc60c899f4b42b75bf9217d3580b1; no voters:
I20260926 20:56:14.027000 25755 raft_consensus.cc:2781] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:14.027479 25755 raft_consensus.cc:686] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [term 1 LEADER]: Becoming Leader. State: Replica: 523a940b1d2e484da650be3827da523f, State: Running, Role: LEADER
I20260926 20:56:14.027974 25755 consensus_queue.cc:237] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } } peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } }
I20260926 20:56:14.032722 25283 catalog_manager.cc:5495] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f reported cstate change: term changed from 0 to 1, leader changed from <none> to 523a940b1d2e484da650be3827da523f (127.8.249.129). New cstate: current_term: 1 leader_uuid: "523a940b1d2e484da650be3827da523f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "523a940b1d2e484da650be3827da523f" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 45715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 } health_report { overall_health: UNKNOWN } } }
I20260926 20:56:14.095403 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.090563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.8.249.129:45715)
0926 20:56:14.090748 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.090752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.090767 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:56:14.090888 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.090892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.090966 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.091097 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:14.091455 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.092173 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.093469 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.093847 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.093935 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.094399 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:14.094933 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:14.095062 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.095140 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.095202 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:56:14.103731 25687 raft_consensus.cc:1261] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb [term 1 FOLLOWER]: Refusing update from remote peer 523a940b1d2e484da650be3827da523f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:14.103756 25556 raft_consensus.cc:1261] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1 [term 1 FOLLOWER]: Refusing update from remote peer 523a940b1d2e484da650be3827da523f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:14.104705 25759 consensus_queue.cc:1035] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcb683d687b7430998ae80a23e1a46eb" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:14.105063 25755 consensus_queue.cc:1035] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dcfc60c899f4b42b75bf9217d3580b1" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:14.115816 25761 mvcc.cc:205] Tried to move back new op lower bound from 7333708489118617600 to 7333708488822099968. Current Snapshot: MvccSnapshot[applied={T|T < 7333708489118617600}]
I20260926 20:56:14.122172 25762 mvcc.cc:205] Tried to move back new op lower bound from 7333708489118617600 to 7333708488822099968. Current Snapshot: MvccSnapshot[applied={T|T < 7333708489118617600}]
I20260926 20:56:14.130087 25284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:14.130380 25284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:14.134095 25284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Sending DeleteTablet for 3 replicas of tablet 2c51fb4243e34246ae4d37b8943845b5
I20260926 20:56:14.135262 25667 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c51fb4243e34246ae4d37b8943845b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:14 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:45980
I20260926 20:56:14.135306 25404 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c51fb4243e34246ae4d37b8943845b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:14 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:40852
I20260926 20:56:14.136083 25536 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c51fb4243e34246ae4d37b8943845b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:14 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:44170
I20260926 20:56:14.137159 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 25321
I20260926 20:56:14.149700 25770 tablet_replica.cc:330] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: stopping tablet replica
I20260926 20:56:14.149945 25772 tablet_replica.cc:330] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1: stopping tablet replica
I20260926 20:56:14.150444 25770 raft_consensus.cc:2229] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:56:14.150518 25772 raft_consensus.cc:2229] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:14.151327 25772 raft_consensus.cc:2258] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:14.151311 25770 raft_consensus.cc:2258] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:14.152079 25771 tablet_replica.cc:330] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb: stopping tablet replica
I20260926 20:56:14.152541 25771 raft_consensus.cc:2229] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:14.153082 25771 raft_consensus.cc:2258] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:14.155467 25771 ts_tablet_manager.cc:1905] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:14.158607 25770 ts_tablet_manager.cc:1905] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:14.158912 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.138899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.8.249.129:45715)
0926 20:56:14.139075 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.139081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.139099 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:56:14.139155 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.139160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.139235 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.139336 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:14.141268 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.142467 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.144674 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.145403 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.145534 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.149983 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:14.149995 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:14.150014 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:14.150059 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:14.155128 (+ 5069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:14.156066 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:14.156077 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:14.156080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:14.156278 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:14.156822 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:14.156830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:14.156833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:14.157281 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:14.157557 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.158200 (+ 643us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.158458 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:56:14.160864 25772 ts_tablet_manager.cc:1905] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:14.161039 25464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:14.164588 25771 ts_tablet_manager.cc:1918] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:14.164814 25771 log.cc:1196] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/2c51fb4243e34246ae4d37b8943845b5
I20260926 20:56:14.165385 25771 ts_tablet_manager.cc:1939] T 2c51fb4243e34246ae4d37b8943845b5 P fcb683d687b7430998ae80a23e1a46eb: Deleting consensus metadata
I20260926 20:56:14.167675 25269 catalog_manager.cc:4841] TS fcb683d687b7430998ae80a23e1a46eb (127.8.249.131:33033): tablet 2c51fb4243e34246ae4d37b8943845b5 (table test-table [id=834e306b7bce4bdd90f2bf250473aea8]) successfully deleted
I20260926 20:56:14.168519 25772 ts_tablet_manager.cc:1918] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:14.168726 25772 log.cc:1196] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/2c51fb4243e34246ae4d37b8943845b5
I20260926 20:56:14.169436 25772 ts_tablet_manager.cc:1939] T 2c51fb4243e34246ae4d37b8943845b5 P 7dcfc60c899f4b42b75bf9217d3580b1: Deleting consensus metadata
I20260926 20:56:14.171449 25271 catalog_manager.cc:4841] TS 7dcfc60c899f4b42b75bf9217d3580b1 (127.8.249.130:39631): tablet 2c51fb4243e34246ae4d37b8943845b5 (table test-table [id=834e306b7bce4bdd90f2bf250473aea8]) successfully deleted
I20260926 20:56:14.269989 25770 ts_tablet_manager.cc:1918] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:14.270303 25770 log.cc:1196] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/2c51fb4243e34246ae4d37b8943845b5
I20260926 20:56:14.270620 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 25321
W20260926 20:56:14.289500 25268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.249.129:45715 (error 108)
I20260926 20:56:14.290230 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 25473
I20260926 20:56:14.291671 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.290849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:45715 (local address 127.0.0.1:40888)
0926 20:56:14.291405 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.291508 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:45715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":364,"threads_started":1}
W20260926 20:56:14.291947 25268 catalog_manager.cc:4584] TS 523a940b1d2e484da650be3827da523f (127.8.249.129:45715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c51fb4243e34246ae4d37b8943845b5: Network error: Client connection negotiation failed: client connection to 127.8.249.129:45715: connect: Connection refused (error 111)
I20260926 20:56:14.304153 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.290998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.8.249.130:39631)
0926 20:56:14.291185 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.291190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.291209 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:56:14.291373 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.291378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.291449 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.291554 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:14.293015 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.293694 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.294994 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.295398 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.295531 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.298097 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:14.298109 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:14.298123 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:14.298158 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:14.301086 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:14.301791 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:14.301799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:14.301802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:14.301934 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:14.302329 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:14.302335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:14.302338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:14.302902 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:14.303197 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.303547 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.303787 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:56:14.304910 25596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:14.345579 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.345111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:45715 (local address 127.0.0.1:40898)
0926 20:56:14.345325 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.345419 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:45715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 20:56:14.346702 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.346325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:45715 (local address 127.0.0.1:40906)
0926 20:56:14.346513 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.346582 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:45715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 20:56:14.399559 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 25473
I20260926 20:56:14.407536 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.407024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:45715 (local address 127.0.0.1:40912)
0926 20:56:14.407276 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.407375 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:45715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 20:56:14.408632 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.408253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:45715 (local address 127.0.0.1:40920)
0926 20:56:14.408412 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.408498 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:45715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:56:14.417440 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 25604
I20260926 20:56:14.430356 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.418225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.8.249.131:33033)
0926 20:56:14.418410 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:14.418415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:14.418433 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:56:14.418571 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:14.418575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:14.418641 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:14.418724 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:14.420183 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.420884 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.422068 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:14.422434 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:14.422530 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:14.425268 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:14.425282 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:14.425298 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:14.425340 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:14.428000 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:14.428621 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:14.428628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:14.428630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:14.428763 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:14.429098 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:14.429101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:14.429103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:14.429429 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:14.429649 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:14.429873 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 20:56:14.430029 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:56:14.431020 25727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:14.489394 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.488811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:45715 (local address 127.0.0.1:40934)
0926 20:56:14.489084 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.489192 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:45715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 20:56:14.490614 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:14.490198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:45715 (local address 127.0.0.1:40940)
0926 20:56:14.490396 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:14.490475 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:45715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 20:56:14.520918 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 25604
I20260926 20:56:14.539072 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 25251
I20260926 20:56:14.540180 25314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:14.652951 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 25251
I20260926 20:56:34.675066 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:42571
--webserver_interface=127.8.249.190
--webserver_port=36095
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:42571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:35.011595 25779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:35.011895 25779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:35.012061 25779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:35.019464 25779 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:56:35.019552 25779 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:56:35.019618 25779 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:56:35.019695 25779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:35.019804 25779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:35.019918 25779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:35.019981 25779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:35.020074 25779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:35.030226 25779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:42571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=36095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:35.031600 25779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:35.033001 25779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:35.045660 25787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:35.045745 25785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:35.045825 25779 server_base.cc:1034] running on GCE node
W20260926 20:56:35.045672 25784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:35.047290 25779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:35.049331 25779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:35.050616 25779 hybrid_clock.cc:648] HybridClock initialized: now 1790456195050544 us; error 48 us; skew 500 ppm
I20260926 20:56:35.051076 25779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:56:35 dist-test-slave-2dg3 krb5kdc[24698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456195, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:35.054608 25779 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:56:35.056162 25779 webserver.cc:460] Webserver started at http://127.8.249.190:36095/ using document root <none> and password file <none>
I20260926 20:56:35.056865 25779 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:35.056964 25779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:35.062667 25779 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:56:35.065641 25794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:35.067013 25779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:35.067153 25779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "8530959d1ef641089b77313224f7c483"
format_stamp: "Formatted at 2026-09-26 20:55:48 on dist-test-slave-2dg3"
server_key: "3a5f1cffc3d0cc27a309bf0796b09d70"
server_key_iv: "f7572c100d40d7c180639a42daca38cb"
server_key_version: "encryptionkey@0"
I20260926 20:56:35.068063 25779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:35.086028 25779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:35.090718 25779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:35.091027 25779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:35.111877 25779 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:42571
I20260926 20:56:35.111896 25845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:42571 every 8 connection(s)
I20260926 20:56:35.113591 25779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:35.119899 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 25779
I20260926 20:56:35.120926 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:45715
--local_ip_for_outbound_sockets=127.8.249.129
--tserver_master_addrs=127.8.249.190:42571
--webserver_port=44111
--webserver_interface=127.8.249.129
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:56:35.121158 25846 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:56:35.126544 25846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Bootstrap starting.
I20260926 20:56:35.148574 25846 log.cc:824] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:35.164625 25846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:56:35.165226 25846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Bootstrap complete.
I20260926 20:56:35.175071 25846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } }
I20260926 20:56:35.176030 25846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8530959d1ef641089b77313224f7c483, State: Initialized, Role: FOLLOWER
I20260926 20:56:35.176712 25846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } }
I20260926 20:56:35.176972 25846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:35.177095 25846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:35.177282 25846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:56:35.181185 25846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } }
I20260926 20:56:35.181661 25846 leader_election.cc:302] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8530959d1ef641089b77313224f7c483; no voters:
I20260926 20:56:35.182363 25846 leader_election.cc:288] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:56:35.182502 25850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:56:35.183748 25850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [term 3 LEADER]: Becoming Leader. State: Replica: 8530959d1ef641089b77313224f7c483, State: Running, Role: LEADER
I20260926 20:56:35.184394 25850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } }
I20260926 20:56:35.184638 25846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:35.187160 25852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8530959d1ef641089b77313224f7c483. Latest consensus state: current_term: 3 leader_uuid: "8530959d1ef641089b77313224f7c483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } } }
I20260926 20:56:35.187601 25852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:35.188115 25851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8530959d1ef641089b77313224f7c483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8530959d1ef641089b77313224f7c483" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 42571 } } }
I20260926 20:56:35.188472 25851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483 [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:35.196784 25856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:35.205739 25856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:35.207325 25856 catalog_manager.cc:1259] Loaded cluster ID: 0c4a1dd3fdcb4fb7898db8f9743038c1
I20260926 20:56:35.207479 25856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:35.219017 25856 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:35.220772 25856 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Loaded TSK: 0
I20260926 20:56:35.221028 25856 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8530959d1ef641089b77313224f7c483: Loaded TSK: 1
F20260926 20:56:35.222295 25856 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f487b9729ad google::LogMessage::Fail() at ??:0
@ 0x7f487b97588a google::LogMessage::SendToLog() at ??:0
@ 0x7f487b9724de google::LogMessage::Flush() at ??:0
@ 0x7f487b973a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f487f6d67d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f487f6c573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f48913a51c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f48913a4783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f48914302a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f48914300c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f489147a5e9 std::function<>::operator()() at ??:0
@ 0x7f48913a8cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f48913a6c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f489142e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f489142e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f487e5be345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f487e5c191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f487e5c1799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f487e59ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f48818766db start_thread at ??:0
@ 0x7f487a80c61f clone at ??:0
W20260926 20:56:35.506016 25848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:35.506325 25848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:35.506479 25848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:35.513761 25848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:35.513907 25848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:35.514091 25848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:56:35.523874 25848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:45715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=44111
--tserver_master_addrs=127.8.249.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:35.525205 25848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:35.526502 25848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:35.539803 25893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:35.539817 25894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:35.539956 25896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:35.540890 25848 server_base.cc:1034] running on GCE node
I20260926 20:56:35.541551 25848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:35.542505 25848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:35.543726 25848 hybrid_clock.cc:648] HybridClock initialized: now 1790456195543682 us; error 59 us; skew 500 ppm
I20260926 20:56:35.544035 25848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:56:35 dist-test-slave-2dg3 krb5kdc[24698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456195, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:35.550928 25848 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:56:35.552662 25848 webserver.cc:460] Webserver started at http://127.8.249.129:44111/ using document root <none> and password file <none>
I20260926 20:56:35.553401 25848 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:35.553580 25848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:35.559437 25848 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:56:35.563848 25903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:35.565284 25848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:56:35.565466 25848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "523a940b1d2e484da650be3827da523f"
format_stamp: "Formatted at 2026-09-26 20:55:49 on dist-test-slave-2dg3"
server_key: "33259942dfb25255b015e0f26ab5f5c0"
server_key_iv: "e52953cbb66b6044cf74ed060ee3b792"
server_key_version: "encryptionkey@0"
I20260926 20:56:35.566524 25848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:35.585713 25848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:35.590368 25848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:35.590684 25848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:35.591787 25848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:35.593751 25910 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260926 20:56:35.599141 25910 ts_tablet_manager.cc:1852] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 20:56:35.600107 25910 ts_tablet_manager.cc:1905] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:35.607088 25910 ts_tablet_manager.cc:1918] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:35.607372 25910 ts_tablet_manager.cc:1939] T 2c51fb4243e34246ae4d37b8943845b5 P 523a940b1d2e484da650be3827da523f: Deleting consensus metadata
I20260926 20:56:35.607868 25848 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 20:56:35.608011 25848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260926 20:56:35.608151 25848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:35.608258 25848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:35.642699 25848 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:45715
I20260926 20:56:35.642748 26016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:45715 every 8 connection(s)
I20260926 20:56:35.644085 25848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:35.649082 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:35.647260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:38029)
0926 20:56:35.648002 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:35.648241 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":333,"threads_started":1}
W20260926 20:56:35.649801 26017 heartbeater.cc:644] Failed to heartbeat to 127.8.249.190:42571 (0 consecutive failures): Network error: Failed to ping master at 127.8.249.190:42571: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
I20260926 20:56:35.650068 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 25848
I20260926 20:56:35.650708 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:35.650326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:42353)
0926 20:56:35.650484 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:35.650569 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:56:35.650877 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:39631
--local_ip_for_outbound_sockets=127.8.249.130
--tserver_master_addrs=127.8.249.190:42571
--webserver_port=35937
--webserver_interface=127.8.249.130
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:56:35.651773 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:35.651314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:49157)
0926 20:56:35.651490 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:35.651590 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 20:56:35.652026 26017 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:56:35.982669 26020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:35.983067 26020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:35.983242 26020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:35.990684 26020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:35.990888 26020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:35.991110 26020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:56:36.001215 26020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:39631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=35937
--tserver_master_addrs=127.8.249.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:36.002635 26020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:36.004055 26020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:36.017664 26025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:36.017633 26026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:36.017704 26028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:36.018667 26020 server_base.cc:1034] running on GCE node
I20260926 20:56:36.019338 26020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:36.020398 26020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:36.021631 26020 hybrid_clock.cc:648] HybridClock initialized: now 1790456196021585 us; error 39 us; skew 500 ppm
I20260926 20:56:36.021972 26020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:56:36 dist-test-slave-2dg3 krb5kdc[24698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456196, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:36.025161 26020 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:56:36.026631 26020 webserver.cc:460] Webserver started at http://127.8.249.130:35937/ using document root <none> and password file <none>
I20260926 20:56:36.027392 26020 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:36.027526 26020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:36.033212 26020 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 20:56:36.036065 26035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:36.037504 26020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:36.037721 26020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "7dcfc60c899f4b42b75bf9217d3580b1"
format_stamp: "Formatted at 2026-09-26 20:55:50 on dist-test-slave-2dg3"
server_key: "689894a0338cb0256f22032a92d89da2"
server_key_iv: "029e986112a53807d5c068e934d66bc8"
server_key_version: "encryptionkey@0"
I20260926 20:56:36.038774 26020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:36.069689 26020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:36.075062 26020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:36.075395 26020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:36.076493 26020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:36.078002 26020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:36.078155 26020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:36.078296 26020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:36.078393 26020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:36.114358 26020 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:39631
I20260926 20:56:36.114413 26147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:39631 every 8 connection(s)
I20260926 20:56:36.116094 26020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:36.116786 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 26020
I20260926 20:56:36.117586 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:33033
--local_ip_for_outbound_sockets=127.8.249.131
--tserver_master_addrs=127.8.249.190:42571
--webserver_port=34841
--webserver_interface=127.8.249.131
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:56:36.122066 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:36.119676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:48405)
0926 20:56:36.120537 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:36.120850 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":316,"threads_started":1}
W20260926 20:56:36.122938 26148 heartbeater.cc:644] Failed to heartbeat to 127.8.249.190:42571 (0 consecutive failures): Network error: Failed to ping master at 127.8.249.190:42571: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
I20260926 20:56:36.124004 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:36.123591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:46923)
0926 20:56:36.123785 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:36.123878 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 20:56:36.125056 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:36.124678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:55753)
0926 20:56:36.124836 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:36.124934 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 20:56:36.125401 26148 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:56:36.455469 26151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:36.455931 26151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:36.456112 26151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:36.465971 26151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:36.466099 26151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:36.466296 26151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:56:36.476652 26151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:33033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=34841
--tserver_master_addrs=127.8.249.190:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:36.478040 26151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:36.479511 26151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:36.492954 26156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:36.492988 26157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:36.493880 26159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:36.494346 26151 server_base.cc:1034] running on GCE node
I20260926 20:56:36.495126 26151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:36.496169 26151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:36.497409 26151 hybrid_clock.cc:648] HybridClock initialized: now 1790456196497363 us; error 42 us; skew 500 ppm
I20260926 20:56:36.497846 26151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:56:36 dist-test-slave-2dg3 krb5kdc[24698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456196, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:36.501003 26151 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:56:36.502596 26151 webserver.cc:460] Webserver started at http://127.8.249.131:34841/ using document root <none> and password file <none>
I20260926 20:56:36.503394 26151 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:36.503552 26151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:36.509136 26151 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 20:56:36.512025 26166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:36.513517 26151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:56:36.513721 26151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "fcb683d687b7430998ae80a23e1a46eb"
format_stamp: "Formatted at 2026-09-26 20:55:50 on dist-test-slave-2dg3"
server_key: "707f4fe479b114311c02e8ede4767d7d"
server_key_iv: "9ac46d7897e9d18f1c74e14673f15e92"
server_key_version: "encryptionkey@0"
I20260926 20:56:36.514755 26151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:36.536554 26151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:36.540997 26151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:36.541293 26151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:36.542347 26151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:36.543939 26151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:36.544070 26151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:36.544206 26151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:36.544296 26151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:36.578365 26151 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:33033
I20260926 20:56:36.578454 26278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:33033 every 8 connection(s)
I20260926 20:56:36.579896 26151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:36.584841 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 26151
I20260926 20:56:36.584862 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:36.582942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:48205)
0926 20:56:36.583686 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:36.583947 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":284,"threads_started":1}
W20260926 20:56:36.585789 26279 heartbeater.cc:644] Failed to heartbeat to 127.8.249.190:42571 (0 consecutive failures): Network error: Failed to ping master at 127.8.249.190:42571: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
I20260926 20:56:36.586669 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:36.586323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:60139)
0926 20:56:36.586485 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:36.586559 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 20:56:36.587553 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:36.587230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:60291)
0926 20:56:36.587369 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:36.587448 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 20:56:36.587886 26279 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:56:36.653434 26283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:36.652588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:44627)
0926 20:56:36.653074 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:36.653159 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":236,"threads_started":1}
I20260926 20:56:37.126924 26284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:37.125991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:53569)
0926 20:56:37.126495 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:37.126597 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":247,"threads_started":1}
I20260926 20:56:37.589293 26285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:37.588504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:44483)
0926 20:56:37.588976 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:37.589069 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":226,"threads_started":1}
I20260926 20:56:37.655288 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:37.654522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:42817)
0926 20:56:37.655024 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:37.655110 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":221,"threads_started":1}
I20260926 20:56:38.128726 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:38.127965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:34139)
0926 20:56:38.128458 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:38.128551 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":238,"threads_started":1}
I20260926 20:56:38.591240 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:38.590404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:48021)
0926 20:56:38.590958 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:38.591048 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":249,"threads_started":1}
I20260926 20:56:38.657142 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:38.656314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:57959)
0926 20:56:38.656867 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:38.656968 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":265,"threads_started":1}
I20260926 20:56:39.130363 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:39.129647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:48377)
0926 20:56:39.130094 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:39.130182 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":236,"threads_started":1}
I20260926 20:56:39.592825 26291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:39.592081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:53049)
0926 20:56:39.592536 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:39.592630 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":236,"threads_started":1}
I20260926 20:56:39.658910 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:39.658046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:43125)
0926 20:56:39.658558 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:39.658655 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":254,"threads_started":1}
I20260926 20:56:40.131917 26293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:40.131244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:42383)
0926 20:56:40.131682 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:40.131759 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":222,"threads_started":1}
I20260926 20:56:40.594357 26294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:40.593669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:33753)
0926 20:56:40.594095 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:40.594179 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":217,"threads_started":1}
I20260926 20:56:40.660681 26295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:40.659925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:44577)
0926 20:56:40.660392 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:40.660497 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":225,"threads_started":1}
I20260926 20:56:41.133512 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:41.132803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:38783)
0926 20:56:41.133267 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:41.133344 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":233,"threads_started":1}
I20260926 20:56:41.595938 26297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:41.595214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:53079)
0926 20:56:41.595669 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:41.595749 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":226,"threads_started":1}
I20260926 20:56:41.662289 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:41.661560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:49821)
0926 20:56:41.662032 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:41.662116 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
I20260926 20:56:42.135452 26299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:42.134549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:56193)
0926 20:56:42.135169 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:42.135287 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":310,"threads_started":1}
I20260926 20:56:42.597616 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:42.596884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:41307)
0926 20:56:42.597352 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:42.597445 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":243,"threads_started":1}
I20260926 20:56:42.664032 26301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:42.663250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:36341)
0926 20:56:42.663758 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:42.663838 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":217,"threads_started":1}
I20260926 20:56:43.137081 26302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:43.136313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:51729)
0926 20:56:43.136788 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:43.136877 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":221,"threads_started":1}
I20260926 20:56:43.599349 26303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:43.598556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:53471)
0926 20:56:43.599059 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:43.599139 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":240,"threads_started":1}
I20260926 20:56:43.665664 26304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:43.664897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:33797)
0926 20:56:43.665407 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:43.665483 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":295,"threads_started":1}
I20260926 20:56:44.138764 26305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.138059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:59251)
0926 20:56:44.138502 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:44.138607 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":231,"threads_started":1}
I20260926 20:56:44.601022 26306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.600311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:44517)
0926 20:56:44.600781 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:44.600874 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":239,"threads_started":1}
I20260926 20:56:44.667325 26307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:44.666572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:39333)
0926 20:56:44.667078 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:44.667171 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":272,"threads_started":1}
I20260926 20:56:45.140480 26308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:45.139730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:35707)
0926 20:56:45.140197 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:45.140309 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":226,"threads_started":1}
I20260926 20:56:45.602873 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:45.601983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:59319)
0926 20:56:45.602534 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:45.602619 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":312,"threads_started":1}
I20260926 20:56:45.668879 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:45.668201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:45767)
0926 20:56:45.668625 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:45.668706 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":220,"threads_started":1}
I20260926 20:56:46.142194 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.141474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:47377)
0926 20:56:46.141916 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:46.142001 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":240,"threads_started":1}
I20260926 20:56:46.604573 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.603862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:59509)
0926 20:56:46.604328 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:46.604410 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":251,"threads_started":1}
I20260926 20:56:46.670552 26313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:46.669735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:53849)
0926 20:56:46.670287 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:46.670365 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":351,"threads_started":1}
I20260926 20:56:47.143863 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:47.143135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:60075)
0926 20:56:47.143574 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:47.143668 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":226,"threads_started":1}
I20260926 20:56:47.606220 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:47.605483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:46475)
0926 20:56:47.605924 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:47.606045 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":229,"threads_started":1}
I20260926 20:56:47.672214 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:47.671451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:48759)
0926 20:56:47.671923 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:47.672020 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":260,"threads_started":1}
I20260926 20:56:48.145481 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:48.144773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:34233)
0926 20:56:48.145232 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:48.145317 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":233,"threads_started":1}
I20260926 20:56:48.608027 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:48.607287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:49885)
0926 20:56:48.607775 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:48.607860 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":262,"threads_started":1}
I20260926 20:56:48.673924 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:48.673116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:39967)
0926 20:56:48.673635 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:48.673750 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":261,"threads_started":1}
I20260926 20:56:49.147449 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:49.146476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:59711)
0926 20:56:49.147088 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:49.147191 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":288,"threads_started":1}
I20260926 20:56:49.609675 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:49.608956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:47903)
0926 20:56:49.609401 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:49.609492 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":217,"threads_started":1}
I20260926 20:56:49.675882 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:49.675013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:52281)
0926 20:56:49.675577 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:49.675676 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":273,"threads_started":1}
I20260926 20:56:50.149096 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:50.148349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:59935)
0926 20:56:50.148827 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:50.148915 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":249,"threads_started":1}
I20260926 20:56:50.611449 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:50.610622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:39023)
0926 20:56:50.611167 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:50.611268 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":307,"threads_started":1}
I20260926 20:56:50.677671 26325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:50.676980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:49883)
0926 20:56:50.677416 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:50.677509 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":212,"threads_started":1}
I20260926 20:56:51.151181 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:51.150342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.130:56611)
0926 20:56:51.150900 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:51.150992 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":277,"threads_started":1}
I20260926 20:56:51.591697 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 25848
I20260926 20:56:51.610064 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:51.593231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.8.249.129:45715)
0926 20:56:51.593858 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:51.593873 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:51.593915 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:56:51.595736 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:51.595745 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:51.596181 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:51.596475 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:51.597849 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:51.599189 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:51.600388 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:51.601048 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:51.601186 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:51.603920 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:51.603927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:51.603943 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:51.603980 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:51.606769 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:51.607461 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:51.607467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:51.607473 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:51.607644 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:51.607956 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:51.607959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:51.607961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:51.608516 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:51.608754 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:51.609294 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 20:56:51.609637 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":330,"threads_started":1}
I20260926 20:56:51.611682 26012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:51.613646 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:51.612607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.131:33563)
0926 20:56:51.613186 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:51.613359 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":241,"threads_started":1}
I20260926 20:56:51.699621 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:51.698544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:42571 (local address 127.8.249.129:42103)
0926 20:56:51.699194 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:51.699373 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.190:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":570,"threads_started":1}
I20260926 20:56:51.703006 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 25848
I20260926 20:56:51.720566 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 26020
I20260926 20:56:51.738996 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:51.721598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.8.249.130:39631)
0926 20:56:51.722207 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:51.722220 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:51.722259 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:56:51.724004 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:51.724023 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:51.724458 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:51.724757 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:51.726134 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:51.727498 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:51.728857 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:51.729562 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:51.729704 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:51.732614 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:51.732622 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:51.732640 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:51.732695 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:51.735793 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:51.736544 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:51.736551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:51.736558 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:51.736745 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:51.737083 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:51.737086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:51.737088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:51.737632 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:51.737873 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:51.738222 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 20:56:51.738587 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":311,"threads_started":1}
I20260926 20:56:51.740473 26143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:51.826759 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 26020
I20260926 20:56:51.844475 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 26151
I20260926 20:56:51.862702 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:51.845460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.8.249.131:33033)
0926 20:56:51.846063 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:51.846078 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:51.846118 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:56:51.847857 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:51.847867 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:51.848340 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:51.848704 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:51.850133 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:51.851445 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:51.852722 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:51.853394 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:51.853594 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:51.856303 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:51.856311 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:51.856328 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:51.856376 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:51.859397 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:51.860083 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:51.860090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:51.860097 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:51.860260 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:51.860660 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:51.860666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:51.860669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:51.861198 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:51.861500 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:51.861982 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 20:56:51.862400 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":272,"threads_started":1}
I20260926 20:56:51.864087 26274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:51.950781 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 26151
2026-09-26T20:56:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64665 ms)
[----------] 30 tests from SecurityITest (295583 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:52 dist-test-slave-2dg3 krb5kdc[26337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:52 dist-test-slave-2dg3 krb5kdc[26337](info): set up 2 sockets
Sep 26 20:56:52 dist-test-slave-2dg3 krb5kdc[26337](info): commencing operation
krb5kdc: starting...
W20260926 20:56:53.051205 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.008s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:56:53 dist-test-slave-2dg3 krb5kdc[26337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:56:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:56:53.306093 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:39095
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:39095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:53.642292 26350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:53.642594 26350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:53.642685 26350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:53.650022 26350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:53.650132 26350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:53.650184 26350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:53.650233 26350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:53.650278 26350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:53.660158 26350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:39095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:39095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:53.661422 26350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:53.662654 26350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:56:53.674655 26350 server_base.cc:1034] running on GCE node
W20260926 20:56:53.674614 26356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:53.674840 26358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:53.674697 26355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:53.675817 26350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:53.677176 26350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:53.678459 26350 hybrid_clock.cc:648] HybridClock initialized: now 1790456213678386 us; error 59 us; skew 500 ppm
I20260926 20:56:53.678862 26350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:56:53 dist-test-slave-2dg3 krb5kdc[26337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456213, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:53.681846 26350 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:56:53.683379 26350 webserver.cc:460] Webserver started at http://127.8.249.190:46533/ using document root <none> and password file <none>
I20260926 20:56:53.684016 26350 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:53.684116 26350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:53.684474 26350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:53.687476 26350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "f84be701dace45518592be6b34e8a4ad"
format_stamp: "Formatted at 2026-09-26 20:56:53 on dist-test-slave-2dg3"
server_key: "b08048396abeaca26e5a6c0117f415ea"
server_key_iv: "af0f1bd3c00c2d6ce80573da1d3127a2"
server_key_version: "encryptionkey@0"
I20260926 20:56:53.688155 26350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "f84be701dace45518592be6b34e8a4ad"
format_stamp: "Formatted at 2026-09-26 20:56:53 on dist-test-slave-2dg3"
server_key: "b08048396abeaca26e5a6c0117f415ea"
server_key_iv: "af0f1bd3c00c2d6ce80573da1d3127a2"
server_key_version: "encryptionkey@0"
I20260926 20:56:53.693125 26350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:56:53.696323 26365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:53.697660 26350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 20:56:53.697832 26350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "f84be701dace45518592be6b34e8a4ad"
format_stamp: "Formatted at 2026-09-26 20:56:53 on dist-test-slave-2dg3"
server_key: "b08048396abeaca26e5a6c0117f415ea"
server_key_iv: "af0f1bd3c00c2d6ce80573da1d3127a2"
server_key_version: "encryptionkey@0"
I20260926 20:56:53.698030 26350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:53.709651 26350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:53.713797 26350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:53.714123 26350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:53.733115 26350 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:39095
I20260926 20:56:53.733139 26416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:39095 every 8 connection(s)
I20260926 20:56:53.734575 26350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:53.738322 26417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:53.740001 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 26350
I20260926 20:56:53.740236 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:56:53.741084 9190 external_mini_cluster.cc:1428] Setting key 9aaa6213409486884470462b3dde3fc0
I20260926 20:56:53.753628 26417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad: Bootstrap starting.
Sep 26 20:56:53 dist-test-slave-2dg3 krb5kdc[26337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:56:53.757028 26417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:53.758340 26417 log.cc:824] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:53.761513 26417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad: No bootstrap required, opened a new log
I20260926 20:56:53.764778 26420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:53.743266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.8.249.190:39095)
0926 20:56:53.744102 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:53.744116 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:53.744158 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:56:53.745798 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:53.745821 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:53.746232 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:53.746566 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:53.748693 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:53.750054 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:53.752167 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:53.752810 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:53.752941 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:53.757293 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:53.757301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:53.757316 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:53.757361 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:53.760585 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:53.761315 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:53.761322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:53.761328 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:53.761505 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:53.761917 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:53.761921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:53.761927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:53.762444 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:53.762761 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:53.763262 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 20:56:53.763749 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":414,"threads_started":1}
I20260926 20:56:53.770653 26417 raft_consensus.cc:348] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84be701dace45518592be6b34e8a4ad" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 39095 } }
I20260926 20:56:53.771126 26417 raft_consensus.cc:374] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:53.771245 26417 raft_consensus.cc:729] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84be701dace45518592be6b34e8a4ad, State: Initialized, Role: FOLLOWER
I20260926 20:56:53.771822 26417 consensus_queue.cc:260] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84be701dace45518592be6b34e8a4ad" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 39095 } }
I20260926 20:56:53.772082 26417 raft_consensus.cc:388] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:53.772233 26417 raft_consensus.cc:482] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:53.772414 26417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:53.775066 26417 raft_consensus.cc:504] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84be701dace45518592be6b34e8a4ad" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 39095 } }
I20260926 20:56:53.775564 26417 leader_election.cc:302] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f84be701dace45518592be6b34e8a4ad; no voters:
I20260926 20:56:53.776216 26417 leader_election.cc:288] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:53.776386 26422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:53.778123 26422 raft_consensus.cc:686] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [term 1 LEADER]: Becoming Leader. State: Replica: f84be701dace45518592be6b34e8a4ad, State: Running, Role: LEADER
I20260926 20:56:53.778641 26422 consensus_queue.cc:237] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84be701dace45518592be6b34e8a4ad" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 39095 } }
I20260926 20:56:53.778831 26417 sys_catalog.cc:564] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:53.781761 26424 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader f84be701dace45518592be6b34e8a4ad. Latest consensus state: current_term: 1 leader_uuid: "f84be701dace45518592be6b34e8a4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84be701dace45518592be6b34e8a4ad" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 39095 } } }
I20260926 20:56:53.782291 26424 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:53.782677 26423 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f84be701dace45518592be6b34e8a4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84be701dace45518592be6b34e8a4ad" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 39095 } } }
I20260926 20:56:53.783038 26423 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:53.784886 26430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:53.790360 26430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:53.800866 26430 catalog_manager.cc:1347] Generated new cluster ID: 31e46c0afc4d4d67991900b48a59d2f4
I20260926 20:56:53.801034 26430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:53.817816 26430 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:53.819298 26430 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:53.835207 26430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad: Generated new TSK 0
I20260926 20:56:53.836140 26430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:56:53.906901 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:39095
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:54.244509 26443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:54.244874 26443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:54.245069 26443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:54.252481 26443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:54.252672 26443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:54.252871 26443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:56:54.262940 26443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:39095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:54.264432 26443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:54.265947 26443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:54.279322 26449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:54.279424 26448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:54.279398 26451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:54.280356 26443 server_base.cc:1034] running on GCE node
I20260926 20:56:54.280956 26443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:54.281924 26443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:54.283159 26443 hybrid_clock.cc:648] HybridClock initialized: now 1790456214283091 us; error 53 us; skew 500 ppm
I20260926 20:56:54.283478 26443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:56:54 dist-test-slave-2dg3 krb5kdc[26337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456214, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:54.286710 26443 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:56:54.288223 26443 webserver.cc:460] Webserver started at http://127.8.249.129:39019/ using document root <none> and password file <none>
I20260926 20:56:54.288898 26443 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:54.289049 26443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:54.289378 26443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:54.292379 26443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "ee50ad1bc8654510b223a753e164b317"
format_stamp: "Formatted at 2026-09-26 20:56:54 on dist-test-slave-2dg3"
server_key: "71b2a2b201e88862f856a50284102bac"
server_key_iv: "cffe64c6c75a48ce7a6e767ebd552330"
server_key_version: "encryptionkey@0"
I20260926 20:56:54.293203 26443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "ee50ad1bc8654510b223a753e164b317"
format_stamp: "Formatted at 2026-09-26 20:56:54 on dist-test-slave-2dg3"
server_key: "71b2a2b201e88862f856a50284102bac"
server_key_iv: "cffe64c6c75a48ce7a6e767ebd552330"
server_key_version: "encryptionkey@0"
I20260926 20:56:54.298717 26443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260926 20:56:54.302479 26458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:54.304036 26443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:54.304220 26443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "ee50ad1bc8654510b223a753e164b317"
format_stamp: "Formatted at 2026-09-26 20:56:54 on dist-test-slave-2dg3"
server_key: "71b2a2b201e88862f856a50284102bac"
server_key_iv: "cffe64c6c75a48ce7a6e767ebd552330"
server_key_version: "encryptionkey@0"
I20260926 20:56:54.304428 26443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:54.322345 26443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:54.327068 26443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:54.327394 26443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:54.328442 26443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:54.329851 26443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:54.329926 26443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:54.330106 26443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:54.330183 26443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:54.364148 26443 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:40507
I20260926 20:56:54.364244 26570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:40507 every 8 connection(s)
I20260926 20:56:54.365533 26443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:54.375036 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 26443
I20260926 20:56:54.375259 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:54.375909 9190 external_mini_cluster.cc:1428] Setting key 5b9888982bc2a248d27c8f28ae3a0186
Sep 26 20:56:54 dist-test-slave-2dg3 krb5kdc[26337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456214, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:56:54.386104 26573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:54.368272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:35923 (local address 127.8.249.190:39095)
0926 20:56:54.368753 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:54.368758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:54.369757 (+ 999us) server_negotiation.cc:407] Connection header received
0926 20:56:54.371691 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:54.371696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:54.371810 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:54.371933 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:54.373406 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:54.374148 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:54.375918 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:54.376401 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:54.376556 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:54.380259 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:54.380265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:54.380268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:54.380299 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:54.382996 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:54.383738 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:54.383744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:54.383754 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:54.383872 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:54.384315 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:54.384320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:54.384323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:54.384610 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:54.384769 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:54.385499 (+ 730us) server_negotiation.cc:299] Negotiation successful
0926 20:56:54.385738 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":265,"threads_started":1}
I20260926 20:56:54.387192 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:54.368859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:39095 (local address 127.8.249.129:35923)
0926 20:56:54.369599 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:54.369635 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:54.371413 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:54.372174 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:54.372182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:54.372680 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:54.373234 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:54.373250 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:54.374321 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:54.374325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:54.375766 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:54.375777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:54.376644 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:54.376653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:54.376898 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:54.377626 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:54.377653 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:54.380020 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:54.383154 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:54.383163 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:54.383180 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:54.383610 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:54.383982 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:54.383988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:54.383991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:54.384197 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:54.384740 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:54.384748 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:54.385114 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 20:56:54.385396 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 20:56:54.385943 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":289,"threads_started":1}
I20260926 20:56:54.388867 26571 heartbeater.cc:347] Connected to a master server at 127.8.249.190:39095
I20260926 20:56:54.389182 26571 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:54.389925 26571 heartbeater.cc:511] Master 127.8.249.190:39095 requested a full tablet report, sending...
I20260926 20:56:54.392125 26382 ts_manager.cc:194] Registered new tserver with Master: ee50ad1bc8654510b223a753e164b317 (127.8.249.129:40507)
I20260926 20:56:54.393693 26382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:35923
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:56:54.429939 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:39095
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:54.764739 26577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:54.765030 26577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:54.765120 26577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:54.772003 26577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:54.772104 26577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:54.772208 26577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:56:54.781827 26577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:39095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:54.783069 26577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:54.784389 26577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:54.797492 26582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:54.797466 26583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:54.797747 26585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:54.799022 26577 server_base.cc:1034] running on GCE node
I20260926 20:56:54.799543 26577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:54.800420 26577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:54.801668 26577 hybrid_clock.cc:648] HybridClock initialized: now 1790456214801597 us; error 68 us; skew 500 ppm
I20260926 20:56:54.802060 26577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:56:54 dist-test-slave-2dg3 krb5kdc[26337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456214, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:54.805244 26577 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:56:54.806710 26577 webserver.cc:460] Webserver started at http://127.8.249.130:39235/ using document root <none> and password file <none>
I20260926 20:56:54.807528 26577 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:54.807673 26577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:54.808039 26577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:54.810982 26577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "d29d6e38f0384e7ca965cd6b057ace09"
format_stamp: "Formatted at 2026-09-26 20:56:54 on dist-test-slave-2dg3"
server_key: "d68107851997314d735fb667d81b52d2"
server_key_iv: "3fc5024b661e7c780194b66c9f379e94"
server_key_version: "encryptionkey@0"
I20260926 20:56:54.811756 26577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "d29d6e38f0384e7ca965cd6b057ace09"
format_stamp: "Formatted at 2026-09-26 20:56:54 on dist-test-slave-2dg3"
server_key: "d68107851997314d735fb667d81b52d2"
server_key_iv: "3fc5024b661e7c780194b66c9f379e94"
server_key_version: "encryptionkey@0"
I20260926 20:56:54.816910 26577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:56:54.820545 26592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:54.821839 26577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:56:54.822028 26577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "d29d6e38f0384e7ca965cd6b057ace09"
format_stamp: "Formatted at 2026-09-26 20:56:54 on dist-test-slave-2dg3"
server_key: "d68107851997314d735fb667d81b52d2"
server_key_iv: "3fc5024b661e7c780194b66c9f379e94"
server_key_version: "encryptionkey@0"
I20260926 20:56:54.822237 26577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:54.865461 26577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:54.869920 26577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:54.870280 26577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:54.871469 26577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:54.872924 26577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:54.873032 26577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:54.873183 26577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:54.873256 26577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:54.908344 26577 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:41465
I20260926 20:56:54.908351 26704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:41465 every 8 connection(s)
I20260926 20:56:54.909695 26577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:54.915277 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 26577
I20260926 20:56:54.915470 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:54.916157 9190 external_mini_cluster.cc:1428] Setting key fcab2daf33bd1b6759759c4df23178f8
Sep 26 20:56:54 dist-test-slave-2dg3 krb5kdc[26337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456214, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:56:54.930841 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:54.912254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:60045 (local address 127.8.249.190:39095)
0926 20:56:54.912728 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:54.912732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:54.913580 (+ 848us) server_negotiation.cc:407] Connection header received
0926 20:56:54.915764 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:54.915768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:54.915846 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:54.915948 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:54.917708 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:54.918493 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:54.920128 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:54.920571 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:54.920713 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:54.924614 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:54.924619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:54.924622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:54.924661 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:54.927648 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:54.928426 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:54.928432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:54.928435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:54.928588 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:54.929034 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:54.929038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:54.929041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:54.929314 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:54.929480 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:54.930182 (+ 702us) server_negotiation.cc:299] Negotiation successful
0926 20:56:54.930409 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":264,"threads_started":1}
I20260926 20:56:54.931845 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:54.912708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:39095 (local address 127.8.249.130:60045)
0926 20:56:54.913379 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:54.913445 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:54.915429 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:54.916233 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:54.916246 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:54.916946 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:54.917529 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:54.917547 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:54.918654 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:54.918659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:54.919969 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:54.919982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:54.920806 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:54.920813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:54.921063 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:54.921807 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:54.921832 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:54.924358 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:54.927799 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:54.927809 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:54.927827 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:54.928289 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:54.928701 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:54.928706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:54.928710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:54.928906 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:54.929441 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:54.929449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:54.929816 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 20:56:54.930086 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 20:56:54.930646 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":278,"threads_started":1}
I20260926 20:56:54.933358 26705 heartbeater.cc:347] Connected to a master server at 127.8.249.190:39095
I20260926 20:56:54.933712 26705 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:54.934602 26705 heartbeater.cc:511] Master 127.8.249.190:39095 requested a full tablet report, sending...
I20260926 20:56:54.936237 26382 ts_manager.cc:194] Registered new tserver with Master: d29d6e38f0384e7ca965cd6b057ace09 (127.8.249.130:41465)
I20260926 20:56:54.937075 26382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:60045
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:56:54.974395 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:39095
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:56:55.315785 26711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:55.316102 26711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:55.316198 26711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:55.323814 26711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:55.323926 26711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:55.324041 26711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:56:55.334260 26711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:39095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:55.335542 26711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:55.336776 26711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:55.349884 26716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:55.349931 26717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:55.350281 26719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:55.351156 26711 server_base.cc:1034] running on GCE node
I20260926 20:56:55.351756 26711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:55.352797 26711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:55.354029 26711 hybrid_clock.cc:648] HybridClock initialized: now 1790456215353975 us; error 35 us; skew 500 ppm
I20260926 20:56:55.354414 26711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:56:55 dist-test-slave-2dg3 krb5kdc[26337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456215, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:56:55.361713 26711 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:56:55.363651 26711 webserver.cc:460] Webserver started at http://127.8.249.131:44717/ using document root <none> and password file <none>
I20260926 20:56:55.364418 26711 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:55.364553 26711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:55.364881 26711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:55.368009 26711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "e66312fef49247ec88865b673733a377"
format_stamp: "Formatted at 2026-09-26 20:56:55 on dist-test-slave-2dg3"
server_key: "84e2393e8fe0cb7ec252036ee6f98242"
server_key_iv: "08bef540ff948d3a807df98bc0687419"
server_key_version: "encryptionkey@0"
I20260926 20:56:55.368819 26711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "e66312fef49247ec88865b673733a377"
format_stamp: "Formatted at 2026-09-26 20:56:55 on dist-test-slave-2dg3"
server_key: "84e2393e8fe0cb7ec252036ee6f98242"
server_key_iv: "08bef540ff948d3a807df98bc0687419"
server_key_version: "encryptionkey@0"
I20260926 20:56:55.374161 26711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 20:56:55.378121 26726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:55.379956 26711 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:56:55.380182 26711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "e66312fef49247ec88865b673733a377"
format_stamp: "Formatted at 2026-09-26 20:56:55 on dist-test-slave-2dg3"
server_key: "84e2393e8fe0cb7ec252036ee6f98242"
server_key_iv: "08bef540ff948d3a807df98bc0687419"
server_key_version: "encryptionkey@0"
I20260926 20:56:55.380456 26711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:55.396929 26571 heartbeater.cc:503] Master 127.8.249.190:39095 was elected leader, sending a full tablet report...
I20260926 20:56:55.407191 26711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:55.411473 26711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:55.411763 26711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:55.412736 26711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:55.414100 26711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:55.414175 26711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:55.414260 26711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:55.414304 26711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:55.448933 26711 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:42317
I20260926 20:56:55.449028 26838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:42317 every 8 connection(s)
I20260926 20:56:55.450269 26711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:55.454608 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 26711
I20260926 20:56:55.454789 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:55.455263 9190 external_mini_cluster.cc:1428] Setting key aec81314a5cae154e8782944ccd3a868
Sep 26 20:56:55 dist-test-slave-2dg3 krb5kdc[26337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456215, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:56:55.470759 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.452826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:32811 (local address 127.8.249.190:39095)
0926 20:56:55.453332 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:55.453339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:55.454275 (+ 936us) server_negotiation.cc:407] Connection header received
0926 20:56:55.456472 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:55.456477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:55.456585 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:55.456711 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:55.458561 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.459442 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.461018 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.461435 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.461587 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.465042 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:55.465048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:55.465056 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:55.465089 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:55.467637 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:55.468347 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:55.468354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:55.468357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:55.468479 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:55.468898 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:55.468904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:55.468907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:55.469181 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:55.469345 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:55.470210 (+ 865us) server_negotiation.cc:299] Negotiation successful
0926 20:56:55.470442 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":226,"threads_started":1}
I20260926 20:56:55.471858 26842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.453394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:39095 (local address 127.8.249.131:32811)
0926 20:56:55.454106 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:55.454150 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:55.456211 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:55.456951 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:55.456960 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:55.457469 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:55.458359 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:55.458388 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.459610 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.459616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:55.460871 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:55.460883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.461677 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.461686 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:55.461938 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.462596 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:56:55.462618 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:55.464822 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:55.467782 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:55.467791 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:55.467808 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:55.468229 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:55.468596 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:56:55.468600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:56:55.468604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:56:55.468786 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:56:55.469323 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:55.469330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:55.469806 (+ 476us) client_negotiation.cc:769] Sending connection context
0926 20:56:55.470112 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 20:56:55.470643 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":273,"threads_started":1}
I20260926 20:56:55.473204 26839 heartbeater.cc:347] Connected to a master server at 127.8.249.190:39095
I20260926 20:56:55.473551 26839 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:55.474274 26839 heartbeater.cc:511] Master 127.8.249.190:39095 requested a full tablet report, sending...
I20260926 20:56:55.475781 26382 ts_manager.cc:194] Registered new tserver with Master: e66312fef49247ec88865b673733a377 (127.8.249.131:42317)
I20260926 20:56:55.476593 26382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:32811
I20260926 20:56:55.483458 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:56:55.501065 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.488172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.8.249.190:39095)
0926 20:56:55.488405 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:55.488414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:55.488759 (+ 345us) server_negotiation.cc:407] Connection header received
0926 20:56:55.488900 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:55.488904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:55.488981 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:55.489069 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:55.490641 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.491418 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.492779 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.493228 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.493377 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.495800 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:55.495805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:55.495808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:55.495844 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:55.498563 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:55.499277 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:55.499281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:55.499283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:55.499414 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:55.499738 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:55.499741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:55.499743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:55.500065 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:55.500247 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:55.500522 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 20:56:55.500695 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 20:56:55.505347 26382 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:55.512681 26382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:55.550735 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.526059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:40507 (local address 127.0.0.1:47246)
0926 20:56:55.538004 (+ 11945us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:55.538084 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:55.538359 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:55.542036 (+ 3677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:55.542044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:55.542079 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:55.542685 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:55.542707 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.544957 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.544963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:55.547468 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:55.547487 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.549413 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.549421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:55.549675 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.549746 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 20:56:55.549901 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 20:56:55.550150 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11537,"mutex_wait_us":203,"thread_start_us":13579,"threads_started":1}
I20260926 20:56:55.550992 26852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.526741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:41465 (local address 127.0.0.1:44920)
0926 20:56:55.539990 (+ 13249us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:55.540010 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:55.540125 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:55.542598 (+ 2473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:55.542602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:55.542630 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:55.543257 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:55.543272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.545232 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.545239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:55.547980 (+ 2741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:55.547998 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.550003 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.550008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:55.550219 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.550251 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 20:56:55.550333 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:56:55.550485 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13089,"mutex_wait_us":98,"thread_start_us":11913,"threads_started":1}
I20260926 20:56:55.551458 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.540540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.8.249.131:42317)
0926 20:56:55.541510 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:55.541519 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:55.541552 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:56:55.541666 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:55.541674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:55.541938 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:55.542137 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:55.543377 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.545106 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.548484 (+ 3378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.549699 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.549867 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.550129 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:55.550875 (+ 746us) server_negotiation.cc:299] Negotiation successful
0926 20:56:55.551100 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":799,"threads_started":1}
I20260926 20:56:55.551677 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.526419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.8.249.129:40507)
0926 20:56:55.541189 (+ 14770us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:55.541205 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:55.541249 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:56:55.541411 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:55.541425 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:55.541798 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:55.542029 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:55.542929 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.544777 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.547662 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.549157 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.549316 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.549566 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:55.551101 (+ 1535us) server_negotiation.cc:299] Negotiation successful
0926 20:56:55.551303 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14520,"thread_start_us":14334,"threads_started":1}
I20260926 20:56:55.552304 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.540090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:42317 (local address 127.0.0.1:53590)
0926 20:56:55.540679 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:55.540699 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:55.540813 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:55.542384 (+ 1571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:55.542391 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:55.542419 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:55.543152 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:55.543162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.545288 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.545293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:55.548294 (+ 3001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:55.548306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.549983 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.549992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:55.550459 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.550488 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 20:56:55.550577 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:56:55.550734 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":90,"thread_start_us":317,"threads_started":1}
I20260926 20:56:55.552435 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.531815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44920 (local address 127.8.249.130:41465)
0926 20:56:55.542006 (+ 10191us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:55.542014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:55.542044 (+ 30us) server_negotiation.cc:407] Connection header received
0926 20:56:55.542137 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:55.542144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:55.542361 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:55.542603 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:55.543500 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.545059 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.548182 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.549741 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.551521 (+ 1780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.551811 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:55.551889 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 20:56:55.552066 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10021,"thread_start_us":9874,"threads_started":1}
I20260926 20:56:55.555944 26640 tablet_service.cc:1467] Processing CreateTablet for tablet 0e703366bf87407dbcec8c3adbd42bbb (DEFAULT_TABLE table=test-table [id=d0c067b7107f4129a78504a7ae0f33c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:55.556211 26774 tablet_service.cc:1467] Processing CreateTablet for tablet 0e703366bf87407dbcec8c3adbd42bbb (DEFAULT_TABLE table=test-table [id=d0c067b7107f4129a78504a7ae0f33c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:55.556211 26506 tablet_service.cc:1467] Processing CreateTablet for tablet 0e703366bf87407dbcec8c3adbd42bbb (DEFAULT_TABLE table=test-table [id=d0c067b7107f4129a78504a7ae0f33c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:55.557317 26640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e703366bf87407dbcec8c3adbd42bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:55.557730 26774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e703366bf87407dbcec8c3adbd42bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:55.557730 26506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e703366bf87407dbcec8c3adbd42bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:55.569309 26858 tablet_bootstrap.cc:492] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09: Bootstrap starting.
I20260926 20:56:55.572723 26858 tablet_bootstrap.cc:654] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:55.573884 26860 tablet_bootstrap.cc:492] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317: Bootstrap starting.
I20260926 20:56:55.575812 26858 log.cc:824] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:55.576115 26859 tablet_bootstrap.cc:492] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377: Bootstrap starting.
I20260926 20:56:55.578608 26860 tablet_bootstrap.cc:654] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:55.579914 26859 tablet_bootstrap.cc:654] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:55.580121 26860 log.cc:824] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:55.581334 26859 log.cc:824] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:55.581912 26858 tablet_bootstrap.cc:492] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09: No bootstrap required, opened a new log
I20260926 20:56:55.582231 26858 ts_tablet_manager.cc:1397] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260926 20:56:55.583701 26860 tablet_bootstrap.cc:492] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317: No bootstrap required, opened a new log
I20260926 20:56:55.584028 26860 ts_tablet_manager.cc:1397] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260926 20:56:55.584561 26859 tablet_bootstrap.cc:492] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377: No bootstrap required, opened a new log
I20260926 20:56:55.584944 26859 ts_tablet_manager.cc:1397] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260926 20:56:55.594188 26859 raft_consensus.cc:348] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee50ad1bc8654510b223a753e164b317" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 40507 } } peers { permanent_uuid: "e66312fef49247ec88865b673733a377" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 42317 } } peers { permanent_uuid: "d29d6e38f0384e7ca965cd6b057ace09" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 41465 } }
I20260926 20:56:55.594676 26859 raft_consensus.cc:374] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:55.594787 26859 raft_consensus.cc:729] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e66312fef49247ec88865b673733a377, State: Initialized, Role: FOLLOWER
I20260926 20:56:55.595235 26858 raft_consensus.cc:348] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee50ad1bc8654510b223a753e164b317" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 40507 } } peers { permanent_uuid: "e66312fef49247ec88865b673733a377" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 42317 } } peers { permanent_uuid: "d29d6e38f0384e7ca965cd6b057ace09" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 41465 } }
I20260926 20:56:55.595741 26858 raft_consensus.cc:374] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:55.595484 26860 raft_consensus.cc:348] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee50ad1bc8654510b223a753e164b317" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 40507 } } peers { permanent_uuid: "e66312fef49247ec88865b673733a377" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 42317 } } peers { permanent_uuid: "d29d6e38f0384e7ca965cd6b057ace09" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 41465 } }
I20260926 20:56:55.595559 26859 consensus_queue.cc:260] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee50ad1bc8654510b223a753e164b317" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 40507 } } peers { permanent_uuid: "e66312fef49247ec88865b673733a377" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 42317 } } peers { permanent_uuid: "d29d6e38f0384e7ca965cd6b057ace09" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 41465 } }
I20260926 20:56:55.595849 26858 raft_consensus.cc:729] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29d6e38f0384e7ca965cd6b057ace09, State: Initialized, Role: FOLLOWER
I20260926 20:56:55.595886 26860 raft_consensus.cc:374] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:55.595985 26860 raft_consensus.cc:729] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee50ad1bc8654510b223a753e164b317, State: Initialized, Role: FOLLOWER
I20260926 20:56:55.596629 26858 consensus_queue.cc:260] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee50ad1bc8654510b223a753e164b317" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 40507 } } peers { permanent_uuid: "e66312fef49247ec88865b673733a377" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 42317 } } peers { permanent_uuid: "d29d6e38f0384e7ca965cd6b057ace09" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 41465 } }
I20260926 20:56:55.596944 26839 heartbeater.cc:503] Master 127.8.249.190:39095 was elected leader, sending a full tablet report...
I20260926 20:56:55.597812 26859 ts_tablet_manager.cc:1428] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377: Time spent starting tablet: real 0.013s user 0.001s sys 0.010s
I20260926 20:56:55.597828 26705 heartbeater.cc:503] Master 127.8.249.190:39095 was elected leader, sending a full tablet report...
I20260926 20:56:55.596635 26860 consensus_queue.cc:260] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee50ad1bc8654510b223a753e164b317" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 40507 } } peers { permanent_uuid: "e66312fef49247ec88865b673733a377" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 42317 } } peers { permanent_uuid: "d29d6e38f0384e7ca965cd6b057ace09" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 41465 } }
I20260926 20:56:55.598973 26858 ts_tablet_manager.cc:1428] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09: Time spent starting tablet: real 0.017s user 0.012s sys 0.005s
I20260926 20:56:55.601183 26860 ts_tablet_manager.cc:1428] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260926 20:56:55.603231 26864 raft_consensus.cc:482] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:55.603523 26864 raft_consensus.cc:504] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee50ad1bc8654510b223a753e164b317" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 40507 } } peers { permanent_uuid: "e66312fef49247ec88865b673733a377" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 42317 } } peers { permanent_uuid: "d29d6e38f0384e7ca965cd6b057ace09" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 41465 } }
I20260926 20:56:55.605760 26864 leader_election.cc:288] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee50ad1bc8654510b223a753e164b317 (127.8.249.129:40507), d29d6e38f0384e7ca965cd6b057ace09 (127.8.249.130:41465)
I20260926 20:56:55.614028 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.606767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:41235 (local address 127.8.249.129:40507)
0926 20:56:55.607104 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:55.607114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:55.607143 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:56:55.607230 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:55.607235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:55.607322 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:55.607438 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:55.607910 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.609004 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.611629 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.612951 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.613145 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.613252 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:55.613606 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 20:56:55.613765 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260926 20:56:55.614053 26842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.606118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:40507 (local address 127.8.249.131:41235)
0926 20:56:55.606343 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:55.606365 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:55.606497 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:55.607476 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:55.607481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:55.607525 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:55.607756 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:55.607770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.609190 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.609196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:55.611416 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:55.611433 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.613123 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.613127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:55.613350 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.613444 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 20:56:55.613551 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 20:56:55.613713 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":134}
I20260926 20:56:55.614182 26867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.606223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:41465 (local address 127.8.249.131:48103)
0926 20:56:55.607512 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:55.607530 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:55.607651 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:55.608003 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:55.608008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:55.608037 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:55.608305 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:55.608312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.609606 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.609611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:55.611667 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:55.611678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.613353 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.613359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:55.613594 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.613624 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 20:56:55.613701 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:56:55.613801 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"mutex_wait_us":2,"thread_start_us":947,"threads_started":1}
I20260926 20:56:55.614185 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.606778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:48103 (local address 127.8.249.130:41465)
0926 20:56:55.607013 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:55.607018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:55.607635 (+ 617us) server_negotiation.cc:407] Connection header received
0926 20:56:55.607769 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:55.607773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:55.607859 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:55.607989 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:55.608506 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.609459 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.611828 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.613153 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.613311 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.613410 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:55.613762 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 20:56:55.613909 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 20:56:55.615291 26660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e703366bf87407dbcec8c3adbd42bbb" candidate_uuid: "e66312fef49247ec88865b673733a377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29d6e38f0384e7ca965cd6b057ace09" is_pre_election: true
I20260926 20:56:55.615286 26526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e703366bf87407dbcec8c3adbd42bbb" candidate_uuid: "e66312fef49247ec88865b673733a377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee50ad1bc8654510b223a753e164b317" is_pre_election: true
I20260926 20:56:55.615814 26526 raft_consensus.cc:2454] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e66312fef49247ec88865b673733a377 in term 0.
I20260926 20:56:55.615823 26660 raft_consensus.cc:2454] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e66312fef49247ec88865b673733a377 in term 0.
I20260926 20:56:55.616576 26729 leader_election.cc:302] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d29d6e38f0384e7ca965cd6b057ace09, e66312fef49247ec88865b673733a377; no voters:
I20260926 20:56:55.617205 26864 raft_consensus.cc:2781] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:55.617347 26864 raft_consensus.cc:482] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:55.617449 26864 raft_consensus.cc:3037] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [term 0 FOLLOWER]: Advancing to term 1
W20260926 20:56:55.617944 26572 tablet.cc:2369] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:55.619824 26864 raft_consensus.cc:504] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee50ad1bc8654510b223a753e164b317" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 40507 } } peers { permanent_uuid: "e66312fef49247ec88865b673733a377" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 42317 } } peers { permanent_uuid: "d29d6e38f0384e7ca965cd6b057ace09" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 41465 } }
I20260926 20:56:55.620529 26864 leader_election.cc:288] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [CANDIDATE]: Term 1 election: Requested vote from peers ee50ad1bc8654510b223a753e164b317 (127.8.249.129:40507), d29d6e38f0384e7ca965cd6b057ace09 (127.8.249.130:41465)
I20260926 20:56:55.621021 26526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e703366bf87407dbcec8c3adbd42bbb" candidate_uuid: "e66312fef49247ec88865b673733a377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee50ad1bc8654510b223a753e164b317"
I20260926 20:56:55.621152 26660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e703366bf87407dbcec8c3adbd42bbb" candidate_uuid: "e66312fef49247ec88865b673733a377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29d6e38f0384e7ca965cd6b057ace09"
I20260926 20:56:55.621256 26526 raft_consensus.cc:3037] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:55.621372 26660 raft_consensus.cc:3037] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:55.624662 26526 raft_consensus.cc:2454] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e66312fef49247ec88865b673733a377 in term 1.
I20260926 20:56:55.624662 26660 raft_consensus.cc:2454] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e66312fef49247ec88865b673733a377 in term 1.
I20260926 20:56:55.625188 26729 leader_election.cc:302] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d29d6e38f0384e7ca965cd6b057ace09, e66312fef49247ec88865b673733a377; no voters:
I20260926 20:56:55.625535 26864 raft_consensus.cc:2781] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:55.625958 26864 raft_consensus.cc:686] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [term 1 LEADER]: Becoming Leader. State: Replica: e66312fef49247ec88865b673733a377, State: Running, Role: LEADER
I20260926 20:56:55.626497 26864 consensus_queue.cc:237] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee50ad1bc8654510b223a753e164b317" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 40507 } } peers { permanent_uuid: "e66312fef49247ec88865b673733a377" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 42317 } } peers { permanent_uuid: "d29d6e38f0384e7ca965cd6b057ace09" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 41465 } }
I20260926 20:56:55.631614 26381 catalog_manager.cc:5495] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 reported cstate change: term changed from 0 to 1, leader changed from <none> to e66312fef49247ec88865b673733a377 (127.8.249.131). New cstate: current_term: 1 leader_uuid: "e66312fef49247ec88865b673733a377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee50ad1bc8654510b223a753e164b317" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 40507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e66312fef49247ec88865b673733a377" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 42317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d29d6e38f0384e7ca965cd6b057ace09" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 41465 } health_report { overall_health: UNKNOWN } } }
I20260926 20:56:55.648766 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.643410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.8.249.131:42317)
0926 20:56:55.643653 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:55.643657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:55.643676 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:56:55.643861 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:55.643865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:55.643949 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:55.644078 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:55.644431 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.645163 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.646500 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.646970 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.647064 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.647619 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:55.648132 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:55.648250 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:55.648367 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 20:56:55.648448 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 20:56:55.657193 26660 raft_consensus.cc:1261] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09 [term 1 FOLLOWER]: Refusing update from remote peer e66312fef49247ec88865b673733a377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:55.657193 26526 raft_consensus.cc:1261] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317 [term 1 FOLLOWER]: Refusing update from remote peer e66312fef49247ec88865b673733a377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:55.658293 26868 consensus_queue.cc:1035] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee50ad1bc8654510b223a753e164b317" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 40507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:55.658648 26864 consensus_queue.cc:1035] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d29d6e38f0384e7ca965cd6b057ace09" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 41465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:56:55.661871 26706 tablet.cc:2369] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:55.671571 26871 mvcc.cc:205] Tried to move back new op lower bound from 7333708659321532416 to 7333708659210428416. Current Snapshot: MvccSnapshot[applied={T|T < 7333708659321532416}]
I20260926 20:56:55.673713 26872 mvcc.cc:205] Tried to move back new op lower bound from 7333708659321532416 to 7333708659210428416. Current Snapshot: MvccSnapshot[applied={T|T < 7333708659321532416}]
I20260926 20:56:55.690160 26380 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:55.690526 26380 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:55.694658 26380 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f84be701dace45518592be6b34e8a4ad: Sending DeleteTablet for 3 replicas of tablet 0e703366bf87407dbcec8c3adbd42bbb
I20260926 20:56:55.695938 26774 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e703366bf87407dbcec8c3adbd42bbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:55 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:53590
I20260926 20:56:55.695931 26506 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e703366bf87407dbcec8c3adbd42bbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:55 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:47246
I20260926 20:56:55.696878 26881 tablet_replica.cc:330] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377: stopping tablet replica
I20260926 20:56:55.697283 26881 raft_consensus.cc:2229] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:56:55.697899 26640 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e703366bf87407dbcec8c3adbd42bbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:55 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:44920
I20260926 20:56:55.698081 26881 raft_consensus.cc:2258] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:55.698319 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 26443
I20260926 20:56:55.707007 26881 ts_tablet_manager.cc:1905] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:55.710687 26882 tablet_replica.cc:330] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317: stopping tablet replica
I20260926 20:56:55.710969 26883 tablet_replica.cc:330] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09: stopping tablet replica
I20260926 20:56:55.711192 26882 raft_consensus.cc:2229] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:55.711373 26883 raft_consensus.cc:2229] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:55.711753 26882 raft_consensus.cc:2258] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:55.711858 26883 raft_consensus.cc:2258] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:55.713964 26882 ts_tablet_manager.cc:1905] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:55.714025 26883 ts_tablet_manager.cc:1905] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:55.720950 26882 ts_tablet_manager.cc:1918] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:55.721154 26882 log.cc:1196] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/0e703366bf87407dbcec8c3adbd42bbb
I20260926 20:56:55.721251 26883 ts_tablet_manager.cc:1918] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:55.721426 26883 log.cc:1196] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/0e703366bf87407dbcec8c3adbd42bbb
I20260926 20:56:55.721714 26882 ts_tablet_manager.cc:1939] T 0e703366bf87407dbcec8c3adbd42bbb P ee50ad1bc8654510b223a753e164b317: Deleting consensus metadata
I20260926 20:56:55.721966 26883 ts_tablet_manager.cc:1939] T 0e703366bf87407dbcec8c3adbd42bbb P d29d6e38f0384e7ca965cd6b057ace09: Deleting consensus metadata
I20260926 20:56:55.723798 26368 catalog_manager.cc:4841] TS d29d6e38f0384e7ca965cd6b057ace09 (127.8.249.130:41465): tablet 0e703366bf87407dbcec8c3adbd42bbb (table test-table [id=d0c067b7107f4129a78504a7ae0f33c0]) successfully deleted
I20260926 20:56:55.724113 26367 catalog_manager.cc:4841] TS ee50ad1bc8654510b223a753e164b317 (127.8.249.129:40507): tablet 0e703366bf87407dbcec8c3adbd42bbb (table test-table [id=d0c067b7107f4129a78504a7ae0f33c0]) successfully deleted
I20260926 20:56:55.724833 26881 ts_tablet_manager.cc:1918] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:55.725001 26881 log.cc:1196] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/0e703366bf87407dbcec8c3adbd42bbb
I20260926 20:56:55.725591 26881 ts_tablet_manager.cc:1939] T 0e703366bf87407dbcec8c3adbd42bbb P e66312fef49247ec88865b673733a377: Deleting consensus metadata
I20260926 20:56:55.727452 26367 catalog_manager.cc:4841] TS e66312fef49247ec88865b673733a377 (127.8.249.131:42317): tablet 0e703366bf87407dbcec8c3adbd42bbb (table test-table [id=d0c067b7107f4129a78504a7ae0f33c0]) successfully deleted
Sep 26 20:56:55 dist-test-slave-2dg3 krb5kdc[26337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:56:55.737736 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.700016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.8.249.129:40507)
0926 20:56:55.700198 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:55.700204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:55.719507 (+ 19303us) server_negotiation.cc:407] Connection header received
0926 20:56:55.720027 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:55.720033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:55.720117 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:55.720196 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:55.722135 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.723314 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.725164 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.725629 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.725780 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.730339 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:55.730353 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:55.730367 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:55.730405 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:55.734076 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:55.735083 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:55.735094 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:55.735098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:55.735261 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:55.735700 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:55.735705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:55.735709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:55.736155 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:55.736465 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:55.737081 (+ 616us) server_negotiation.cc:299] Negotiation successful
0926 20:56:55.737337 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:56:55.739333 26566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:55.838335 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 26443
I20260926 20:56:55.856496 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 26577
Sep 26 20:56:55 dist-test-slave-2dg3 krb5kdc[26337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:56:55.874503 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.857380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.8.249.130:41465)
0926 20:56:55.857596 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:55.857602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:55.857628 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:56:55.857728 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:55.857735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:55.857818 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:55.857955 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:55.859278 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.860437 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.861900 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.862541 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.862683 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:55.867598 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:55.867619 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:55.867642 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:55.867694 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:55.871497 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:55.872282 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:55.872293 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:55.872298 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:55.872448 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:55.872796 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:55.872801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:55.872806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:55.873288 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:55.873577 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:55.873869 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 20:56:55.874105 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:56:55.875298 26700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:55.964893 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 26577
I20260926 20:56:55.989223 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 26711
Sep 26 20:56:55 dist-test-slave-2dg3 krb5kdc[26337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:56:56.008510 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:55.990221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.8.249.131:42317)
0926 20:56:55.990472 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:55.990479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:55.990603 (+ 124us) server_negotiation.cc:407] Connection header received
0926 20:56:55.990779 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:55.990785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:55.990939 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:55.991058 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:55.992401 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.993732 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.995353 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:55.996077 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:55.996219 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:56.000853 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:56.000876 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:56.000897 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:56:56.000959 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:56.005331 (+ 4372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:56.006161 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:56.006171 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:56.006175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:56.006394 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:56:56.006767 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:56:56.006772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:56:56.006776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:56:56.007305 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:56.007662 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:56.007833 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 20:56:56.008080 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 20:56:56.009374 26834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:56.133656 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 26711
I20260926 20:56:56.153162 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 26350
I20260926 20:56:56.154314 26412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:56.265992 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 26350
2026-09-26T20:56:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4322 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T20:56:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:56:56Z Disabled control of system clock
I20260926 20:56:56.348117 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:46363
--webserver_interface=127.8.249.190
--webserver_port=0
--builtin_ntp_servers=127.8.249.148:35781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:46363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:56:56.699015 26892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:56.699370 26892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:56.699460 26892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:56.707340 26892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:56:56.707495 26892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:56.707540 26892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:56.707597 26892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:56:56.707648 26892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:56:56.717715 26892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:35781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:46363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:46363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:56.719120 26892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:56.721007 26892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:56.734129 26897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:56.733906 26900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:56.733997 26898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:56.735224 26892 server_base.cc:1034] running on GCE node
I20260926 20:56:56.736125 26892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:56.738011 26892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:56.739248 26892 hybrid_clock.cc:648] HybridClock initialized: now 1790456216739201 us; error 42 us; skew 500 ppm
I20260926 20:56:56.739622 26892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:56:56.741976 26892 webserver.cc:460] Webserver started at http://127.8.249.190:45251/ using document root <none> and password file <none>
I20260926 20:56:56.742645 26892 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:56.742731 26892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:56.743095 26892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:56.746070 26892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "21574a097e4d4994a3f9ba5c39cf4c3e"
format_stamp: "Formatted at 2026-09-26 20:56:56 on dist-test-slave-2dg3"
server_key: "d673ce3bf98daedef3947211e0c60a97"
server_key_iv: "2d44457e4186638d066a12d8e619a951"
server_key_version: "encryptionkey@0"
I20260926 20:56:56.746862 26892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "21574a097e4d4994a3f9ba5c39cf4c3e"
format_stamp: "Formatted at 2026-09-26 20:56:56 on dist-test-slave-2dg3"
server_key: "d673ce3bf98daedef3947211e0c60a97"
server_key_iv: "2d44457e4186638d066a12d8e619a951"
server_key_version: "encryptionkey@0"
I20260926 20:56:56.751901 26892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 20:56:56.755541 26906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:56.757045 26892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:56:56.757215 26892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "21574a097e4d4994a3f9ba5c39cf4c3e"
format_stamp: "Formatted at 2026-09-26 20:56:56 on dist-test-slave-2dg3"
server_key: "d673ce3bf98daedef3947211e0c60a97"
server_key_iv: "2d44457e4186638d066a12d8e619a951"
server_key_version: "encryptionkey@0"
I20260926 20:56:56.757404 26892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:56.777002 26892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:56.777817 26892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:56.778070 26892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:56.797164 26892 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:46363
I20260926 20:56:56.797188 26957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:46363 every 8 connection(s)
I20260926 20:56:56.798607 26892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:56:56.802556 26958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:56.804773 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 26892
I20260926 20:56:56.804955 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:56:56.805517 9190 external_mini_cluster.cc:1428] Setting key fc59e411d3a784f4d9be583bcaec20bd
I20260926 20:56:56.814941 26958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e: Bootstrap starting.
I20260926 20:56:56.819219 26958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:56.820524 26961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:56.807766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.8.249.190:46363)
0926 20:56:56.808846 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:56.808858 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:56.808910 (+ 52us) server_negotiation.cc:407] Connection header received
0926 20:56:56.810661 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:56.810684 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:56.811231 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:56.811650 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:56.812230 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:56.813547 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:56.814946 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:56.816003 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:56.816203 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:56.817464 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:56.817474 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:56.817492 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:56.818223 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:56.818349 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:56.818378 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:56.818503 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:56.818738 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 20:56:56.819153 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":371,"threads_started":1}
I20260926 20:56:56.820566 26958 log.cc:824] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:56.823918 26958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e: No bootstrap required, opened a new log
I20260926 20:56:56.831871 26958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21574a097e4d4994a3f9ba5c39cf4c3e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46363 } }
I20260926 20:56:56.832295 26958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:56.832366 26958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21574a097e4d4994a3f9ba5c39cf4c3e, State: Initialized, Role: FOLLOWER
I20260926 20:56:56.832985 26958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21574a097e4d4994a3f9ba5c39cf4c3e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46363 } }
I20260926 20:56:56.833197 26958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:56:56.833304 26958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:56:56.833431 26958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:56.835623 26958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21574a097e4d4994a3f9ba5c39cf4c3e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46363 } }
I20260926 20:56:56.836270 26958 leader_election.cc:302] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21574a097e4d4994a3f9ba5c39cf4c3e; no voters:
I20260926 20:56:56.836953 26958 leader_election.cc:288] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:56:56.837081 26963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:56.837822 26963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [term 1 LEADER]: Becoming Leader. State: Replica: 21574a097e4d4994a3f9ba5c39cf4c3e, State: Running, Role: LEADER
I20260926 20:56:56.838322 26963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21574a097e4d4994a3f9ba5c39cf4c3e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46363 } }
I20260926 20:56:56.839164 26958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:56:56.841219 26965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21574a097e4d4994a3f9ba5c39cf4c3e. Latest consensus state: current_term: 1 leader_uuid: "21574a097e4d4994a3f9ba5c39cf4c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21574a097e4d4994a3f9ba5c39cf4c3e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46363 } } }
I20260926 20:56:56.841784 26965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:56.841638 26964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21574a097e4d4994a3f9ba5c39cf4c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21574a097e4d4994a3f9ba5c39cf4c3e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46363 } } }
I20260926 20:56:56.841986 26964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e [sys.catalog]: This master's current role is: LEADER
I20260926 20:56:56.844506 26969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:56:56.850399 26969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:56:56.860293 26969 catalog_manager.cc:1347] Generated new cluster ID: 07bf9157788e475881f80a2821706599
I20260926 20:56:56.860433 26969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:56:56.868549 26969 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:56:56.869925 26969 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:56:56.879472 26969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e: Generated new TSK 0
I20260926 20:56:56.880646 26969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:56:56.891260 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46363
--builtin_ntp_servers=127.8.249.148:35781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:56:57.240125 26982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:57.240504 26982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:57.240669 26982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:57.248358 26982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:57.248476 26982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:57.248653 26982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:56:57.258069 26982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:35781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:57.259475 26982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:57.261283 26982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:57.275377 26988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:57.275454 26987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:57.275488 26990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:57.276172 26982 server_base.cc:1034] running on GCE node
I20260926 20:56:57.276842 26982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:57.277915 26982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:57.279189 26982 hybrid_clock.cc:648] HybridClock initialized: now 1790456217279122 us; error 63 us; skew 500 ppm
I20260926 20:56:57.279573 26982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:56:57.282133 26982 webserver.cc:460] Webserver started at http://127.8.249.129:45745/ using document root <none> and password file <none>
I20260926 20:56:57.283016 26982 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:57.283171 26982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:57.283522 26982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:57.287146 26982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "8752865be9ad439ebb52d162fd73c205"
format_stamp: "Formatted at 2026-09-26 20:56:57 on dist-test-slave-2dg3"
server_key: "c048e256315a833e1f5c5a6f0688177a"
server_key_iv: "e575cd78153d95c857bd0244e68a4ba2"
server_key_version: "encryptionkey@0"
I20260926 20:56:57.288023 26982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "8752865be9ad439ebb52d162fd73c205"
format_stamp: "Formatted at 2026-09-26 20:56:57 on dist-test-slave-2dg3"
server_key: "c048e256315a833e1f5c5a6f0688177a"
server_key_iv: "e575cd78153d95c857bd0244e68a4ba2"
server_key_version: "encryptionkey@0"
I20260926 20:56:57.293521 26982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 20:56:57.297338 26996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:57.299062 26982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 20:56:57.299268 26982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "8752865be9ad439ebb52d162fd73c205"
format_stamp: "Formatted at 2026-09-26 20:56:57 on dist-test-slave-2dg3"
server_key: "c048e256315a833e1f5c5a6f0688177a"
server_key_iv: "e575cd78153d95c857bd0244e68a4ba2"
server_key_version: "encryptionkey@0"
I20260926 20:56:57.299515 26982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:57.325960 26982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:57.326898 26982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:57.327225 26982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:57.328285 26982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:57.329932 26982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:57.330024 26982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:57.330108 26982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:57.330169 26982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:57.364188 26982 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:39067
I20260926 20:56:57.364249 27108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:39067 every 8 connection(s)
I20260926 20:56:57.365643 26982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:56:57.366945 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 26982
I20260926 20:56:57.367128 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:56:57.367702 9190 external_mini_cluster.cc:1428] Setting key ea62c87c1b70a914357670452ca23d50
I20260926 20:56:57.371224 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46363
--builtin_ntp_servers=127.8.249.148:35781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:56:57.380014 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:57.368870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:60583 (local address 127.8.249.190:46363)
0926 20:56:57.369896 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:57.369918 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:57.371020 (+ 1102us) server_negotiation.cc:407] Connection header received
0926 20:56:57.373026 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:57.373032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:57.373137 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:57.373263 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:57.374455 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:57.375286 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:57.376917 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:57.377328 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:57.377482 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:57.379122 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:57.379127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:57.379130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:57.379150 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:57.379193 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:57.379206 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:57.379281 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:57.379552 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 20:56:57.379688 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":733,"threads_started":1}
I20260926 20:56:57.381600 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:57.369703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46363 (local address 127.8.249.129:60583)
0926 20:56:57.370771 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:57.370899 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:57.372709 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:57.373533 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:57.373542 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:57.373648 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:57.374258 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:57.374275 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:57.375466 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:57.375471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:57.376756 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:57.376769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:57.377590 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:57.377597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:57.377836 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:57.378662 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:57.378668 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:57.378733 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:57.378895 (+ 162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:57.379314 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:57.379328 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:57.379340 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 20:56:57.379537 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 20:56:57.380019 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":378,"threads_started":1}
I20260926 20:56:57.383385 27109 heartbeater.cc:347] Connected to a master server at 127.8.249.190:46363
I20260926 20:56:57.383750 27109 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:57.384548 27109 heartbeater.cc:511] Master 127.8.249.190:46363 requested a full tablet report, sending...
I20260926 20:56:57.386946 26923 ts_manager.cc:194] Registered new tserver with Master: 8752865be9ad439ebb52d162fd73c205 (127.8.249.129:39067)
I20260926 20:56:57.389137 26923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.249.129:60583
W20260926 20:56:57.708988 27113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:57.709299 27113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:57.709395 27113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:57.716663 27113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:57.716792 27113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:57.716900 27113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:56:57.726174 27113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:35781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:57.727481 27113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:57.729163 27113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:57.742552 27118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:57.742656 27121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:57.742624 27119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:57.743693 27113 server_base.cc:1034] running on GCE node
I20260926 20:56:57.744431 27113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:57.745582 27113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:57.746902 27113 hybrid_clock.cc:648] HybridClock initialized: now 1790456217746784 us; error 58 us; skew 500 ppm
I20260926 20:56:57.747236 27113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:56:57.749666 27113 webserver.cc:460] Webserver started at http://127.8.249.130:39789/ using document root <none> and password file <none>
I20260926 20:56:57.750380 27113 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:57.750512 27113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:57.750923 27113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:57.753831 27113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10"
format_stamp: "Formatted at 2026-09-26 20:56:57 on dist-test-slave-2dg3"
server_key: "7afa24b75ee79bdfe2e6e15a25c8a7be"
server_key_iv: "1dfff986776e61a03be00e7500f03d1b"
server_key_version: "encryptionkey@0"
I20260926 20:56:57.754667 27113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10"
format_stamp: "Formatted at 2026-09-26 20:56:57 on dist-test-slave-2dg3"
server_key: "7afa24b75ee79bdfe2e6e15a25c8a7be"
server_key_iv: "1dfff986776e61a03be00e7500f03d1b"
server_key_version: "encryptionkey@0"
I20260926 20:56:57.760017 27113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 20:56:57.763722 27127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:57.764973 27113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:56:57.765154 27113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10"
format_stamp: "Formatted at 2026-09-26 20:56:57 on dist-test-slave-2dg3"
server_key: "7afa24b75ee79bdfe2e6e15a25c8a7be"
server_key_iv: "1dfff986776e61a03be00e7500f03d1b"
server_key_version: "encryptionkey@0"
I20260926 20:56:57.765383 27113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:57.789033 27113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:57.789903 27113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:57.790225 27113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:57.791304 27113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:57.792759 27113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:57.792897 27113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:57.793020 27113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:57.793108 27113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:57.827735 27113 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:37307
I20260926 20:56:57.827832 27239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:37307 every 8 connection(s)
I20260926 20:56:57.828994 27113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:56:57.829903 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 27113
I20260926 20:56:57.830086 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:56:57.830693 9190 external_mini_cluster.cc:1428] Setting key 50d00e9d74cdb1f5c8cccb700fe28d94
I20260926 20:56:57.834215 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46363
--builtin_ntp_servers=127.8.249.148:35781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:56:57.847178 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:57.832012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:51279 (local address 127.8.249.190:46363)
0926 20:56:57.832238 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:57.832242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:57.835628 (+ 3386us) server_negotiation.cc:407] Connection header received
0926 20:56:57.838554 (+ 2926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:57.838559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:57.838634 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:57.838728 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:57.840477 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:57.841288 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:57.843936 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:57.844532 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:57.844713 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:57.846277 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:57.846282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:57.846286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:57.846305 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:57.846335 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:57.846347 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:57.846458 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:57.846700 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 20:56:57.846832 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 20:56:57.848261 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:57.832753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46363 (local address 127.8.249.130:51279)
0926 20:56:57.835428 (+ 2675us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:57.835487 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:57.838207 (+ 2720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:57.839066 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:57.839080 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:57.839227 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:57.840264 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:57.840296 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:57.841454 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:57.841462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:57.843726 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:57.843739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:57.844808 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:57.844814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:57.845169 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:57.845937 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:57.845939 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:57.845980 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:57.846065 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:57.846468 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:57.846480 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:57.846490 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:56:57.846674 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 20:56:57.847136 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2030,"thread_start_us":1619,"threads_started":1}
I20260926 20:56:57.850016 27240 heartbeater.cc:347] Connected to a master server at 127.8.249.190:46363
I20260926 20:56:57.850538 27240 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:57.851593 27240 heartbeater.cc:511] Master 127.8.249.190:46363 requested a full tablet report, sending...
I20260926 20:56:57.853235 26923 ts_manager.cc:194] Registered new tserver with Master: 2c3e9ccbf0fd49d39c86fc44b2bbba10 (127.8.249.130:37307)
I20260926 20:56:57.854486 26923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.249.130:51279
W20260926 20:56:58.175832 27243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:56:58.176126 27243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:56:58.176216 27243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:56:58.183560 27243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:56:58.183673 27243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:56:58.183786 27243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:56:58.192847 27243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:35781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:46363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:56:58.194104 27243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:56:58.196054 27243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:56:58.210001 27248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:58.209962 27251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:56:58.210057 27249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:56:58.211236 27243 server_base.cc:1034] running on GCE node
I20260926 20:56:58.211827 27243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:56:58.212927 27243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:56:58.214141 27243 hybrid_clock.cc:648] HybridClock initialized: now 1790456218214077 us; error 64 us; skew 500 ppm
I20260926 20:56:58.214473 27243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:56:58.216863 27243 webserver.cc:460] Webserver started at http://127.8.249.131:35059/ using document root <none> and password file <none>
I20260926 20:56:58.217592 27243 fs_manager.cc:359] Metadata directory not provided
I20260926 20:56:58.217710 27243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:56:58.218057 27243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:56:58.221028 27243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "f06f1d5c93fe49298b0535fb0f1a4d20"
format_stamp: "Formatted at 2026-09-26 20:56:58 on dist-test-slave-2dg3"
server_key: "c2f7449ab6c88117e768f2509933c779"
server_key_iv: "c430aba93ec761f5cb7744b2734b8509"
server_key_version: "encryptionkey@0"
I20260926 20:56:58.221817 27243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "f06f1d5c93fe49298b0535fb0f1a4d20"
format_stamp: "Formatted at 2026-09-26 20:56:58 on dist-test-slave-2dg3"
server_key: "c2f7449ab6c88117e768f2509933c779"
server_key_iv: "c430aba93ec761f5cb7744b2734b8509"
server_key_version: "encryptionkey@0"
I20260926 20:56:58.227125 27243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 20:56:58.230544 27257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:58.231974 27243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:56:58.232157 27243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "f06f1d5c93fe49298b0535fb0f1a4d20"
format_stamp: "Formatted at 2026-09-26 20:56:58 on dist-test-slave-2dg3"
server_key: "c2f7449ab6c88117e768f2509933c779"
server_key_iv: "c430aba93ec761f5cb7744b2734b8509"
server_key_version: "encryptionkey@0"
I20260926 20:56:58.232388 27243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:56:58.259416 27243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:56:58.260346 27243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:56:58.260700 27243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:56:58.261860 27243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:56:58.263710 27243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:56:58.263829 27243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:58.263969 27243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:56:58.264056 27243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:56:58.297960 27243 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:37483
I20260926 20:56:58.298014 27369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:37483 every 8 connection(s)
I20260926 20:56:58.299337 27243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:56:58.304469 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 27243
I20260926 20:56:58.304657 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:56:58.305089 9190 external_mini_cluster.cc:1428] Setting key e8dd6eb09ce2ab3dcd42d87ab319ed53
I20260926 20:56:58.312071 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.301893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:42495 (local address 127.8.249.190:46363)
0926 20:56:58.302081 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:58.302086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:58.303598 (+ 1512us) server_negotiation.cc:407] Connection header received
0926 20:56:58.305697 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:58.305703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:58.305799 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:58.305925 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:58.307287 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.308133 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.309557 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.309959 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.310047 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.311310 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:58.311315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:58.311318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:58.311335 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:58.311390 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:58.311403 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:58.311498 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:58.311748 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 20:56:58.311825 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:56:58.312980 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.302397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46363 (local address 127.8.249.131:42495)
0926 20:56:58.303442 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:58.303495 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:58.305426 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:58.306297 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:58.306305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:58.306385 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:56:58.307104 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:58.307133 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.308272 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.308276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:58.309437 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:58.309447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.310131 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.310135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:58.310310 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.311001 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:56:58.311004 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:56:58.311043 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:56:58.311111 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:56:58.311523 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:56:58.311530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:56:58.311536 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:56:58.311671 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 20:56:58.311976 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":320,"threads_started":1}
I20260926 20:56:58.314325 27370 heartbeater.cc:347] Connected to a master server at 127.8.249.190:46363
I20260926 20:56:58.314584 27370 heartbeater.cc:464] Registering TS with master...
I20260926 20:56:58.315297 27370 heartbeater.cc:511] Master 127.8.249.190:46363 requested a full tablet report, sending...
I20260926 20:56:58.316638 26923 ts_manager.cc:194] Registered new tserver with Master: f06f1d5c93fe49298b0535fb0f1a4d20 (127.8.249.131:37483)
I20260926 20:56:58.317384 26923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.249.131:42495
I20260926 20:56:58.320309 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:56:58.330864 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.324618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.8.249.190:46363)
0926 20:56:58.324865 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:58.324874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:58.325183 (+ 309us) server_negotiation.cc:407] Connection header received
0926 20:56:58.325410 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:58.325415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:58.325486 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:58.325592 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:58.326099 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.326871 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.328418 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.328836 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.328963 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.330079 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:58.330084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:58.330087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:58.330103 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:58.330134 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:58.330146 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:58.330241 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:58.330395 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 20:56:58.330488 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:56:58.334393 26923 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:56:58.341711 26923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:56:58.376560 27384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.354487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.8.249.130:37307)
0926 20:56:58.365656 (+ 11169us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:58.365669 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:58.366898 (+ 1229us) server_negotiation.cc:407] Connection header received
0926 20:56:58.367023 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:58.367031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:58.367292 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:58.367494 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:58.368566 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.370275 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.373840 (+ 3565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.375288 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.375470 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.375709 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:58.375955 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 20:56:58.376145 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10971,"thread_start_us":11182,"threads_started":1}
I20260926 20:56:58.377352 27383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.354277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:37307 (local address 127.0.0.1:33286)
0926 20:56:58.366391 (+ 12114us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:58.366441 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:58.366669 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:58.367803 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:58.367815 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:58.367861 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:58.368356 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:58.368370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.370426 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.370432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:58.373615 (+ 3183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:58.373632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.375491 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.375498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:58.375695 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.375752 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 20:56:58.376767 (+ 1015us) client_negotiation.cc:240] Negotiation successful
0926 20:56:58.376959 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11854,"mutex_wait_us":351,"thread_start_us":15033,"threads_started":1}
I20260926 20:56:58.378003 27382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.354445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:37483 (local address 127.0.0.1:45658)
0926 20:56:58.368027 (+ 13582us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:58.368055 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:58.368235 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:58.369348 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:58.369356 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:58.369426 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:58.369950 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:58.369965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.372514 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.372519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:58.374755 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:58.374771 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.376610 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.376616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:58.377203 (+ 587us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.377239 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 20:56:58.377356 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:56:58.377563 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13402,"mutex_wait_us":44,"thread_start_us":11577,"threads_started":1}
I20260926 20:56:58.378310 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.355030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.8.249.131:37483)
0926 20:56:58.367288 (+ 12258us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:58.367300 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:58.368413 (+ 1113us) server_negotiation.cc:407] Connection header received
0926 20:56:58.368543 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:58.368554 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:58.368810 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:58.369018 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:58.370213 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.372318 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.375014 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.376387 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.376540 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.376778 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:58.377425 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 20:56:58.377623 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12022,"thread_start_us":12262,"threads_started":1}
I20260926 20:56:58.381803 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.357006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39067 (local address 127.0.0.1:43850)
0926 20:56:58.372644 (+ 15638us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:58.372663 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:58.372779 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:58.374141 (+ 1362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:58.374146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:58.374171 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:58.374537 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:58.374545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.377044 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.377050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:58.379276 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:58.379291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.381051 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.381057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:58.381229 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.381263 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:56:58.381360 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 20:56:58.381495 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15483,"thread_start_us":15853,"threads_started":1}
I20260926 20:56:58.381935 27387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.357208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43850 (local address 127.8.249.129:39067)
0926 20:56:58.373595 (+ 16387us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:58.373604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:58.373632 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:56:58.373724 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:58.373734 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:58.373950 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:58.374148 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:58.374723 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.376837 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.379476 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.380856 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.381007 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.381263 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:58.381400 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 20:56:58.381571 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16208,"thread_start_us":16091,"threads_started":1}
I20260926 20:56:58.381647 27175 tablet_service.cc:1467] Processing CreateTablet for tablet 2d12766803234942827144de85e467da (DEFAULT_TABLE table=test-table [id=cc07e056a8b946a5a833773cce2d9695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:58.381690 27305 tablet_service.cc:1467] Processing CreateTablet for tablet 2d12766803234942827144de85e467da (DEFAULT_TABLE table=test-table [id=cc07e056a8b946a5a833773cce2d9695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:58.383184 27305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d12766803234942827144de85e467da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:58.383186 27175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d12766803234942827144de85e467da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:58.385147 27044 tablet_service.cc:1467] Processing CreateTablet for tablet 2d12766803234942827144de85e467da (DEFAULT_TABLE table=test-table [id=cc07e056a8b946a5a833773cce2d9695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:56:58.386235 27044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d12766803234942827144de85e467da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:56:58.392256 27109 heartbeater.cc:503] Master 127.8.249.190:46363 was elected leader, sending a full tablet report...
I20260926 20:56:58.397750 27388 tablet_bootstrap.cc:492] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205: Bootstrap starting.
I20260926 20:56:58.399533 27389 tablet_bootstrap.cc:492] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10: Bootstrap starting.
I20260926 20:56:58.401244 27390 tablet_bootstrap.cc:492] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20: Bootstrap starting.
I20260926 20:56:58.402954 27388 tablet_bootstrap.cc:654] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:58.403204 27389 tablet_bootstrap.cc:654] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:58.404484 27389 log.cc:824] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:58.404502 27388 log.cc:824] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:58.408457 27389 tablet_bootstrap.cc:492] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10: No bootstrap required, opened a new log
I20260926 20:56:58.408826 27389 ts_tablet_manager.cc:1397] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 20:56:58.410610 27390 tablet_bootstrap.cc:654] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20: Neither blocks nor log segments found. Creating new log.
I20260926 20:56:58.412606 27390 log.cc:824] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20: Log is configured to *not* fsync() on all Append() calls
I20260926 20:56:58.414075 27388 tablet_bootstrap.cc:492] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205: No bootstrap required, opened a new log
I20260926 20:56:58.414382 27388 ts_tablet_manager.cc:1397] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.006s
I20260926 20:56:58.415681 27390 tablet_bootstrap.cc:492] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20: No bootstrap required, opened a new log
I20260926 20:56:58.415942 27390 ts_tablet_manager.cc:1397] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260926 20:56:58.417987 27389 raft_consensus.cc:348] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 37307 } } peers { permanent_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37483 } } peers { permanent_uuid: "8752865be9ad439ebb52d162fd73c205" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39067 } }
I20260926 20:56:58.418355 27389 raft_consensus.cc:374] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:58.418481 27389 raft_consensus.cc:729] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3e9ccbf0fd49d39c86fc44b2bbba10, State: Initialized, Role: FOLLOWER
I20260926 20:56:58.419112 27389 consensus_queue.cc:260] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 37307 } } peers { permanent_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37483 } } peers { permanent_uuid: "8752865be9ad439ebb52d162fd73c205" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39067 } }
I20260926 20:56:58.420192 27240 heartbeater.cc:503] Master 127.8.249.190:46363 was elected leader, sending a full tablet report...
I20260926 20:56:58.420832 27389 ts_tablet_manager.cc:1428] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260926 20:56:58.425292 27388 raft_consensus.cc:348] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 37307 } } peers { permanent_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37483 } } peers { permanent_uuid: "8752865be9ad439ebb52d162fd73c205" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39067 } }
I20260926 20:56:58.425714 27388 raft_consensus.cc:374] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:58.425875 27388 raft_consensus.cc:729] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8752865be9ad439ebb52d162fd73c205, State: Initialized, Role: FOLLOWER
I20260926 20:56:58.426246 27390 raft_consensus.cc:348] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 37307 } } peers { permanent_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37483 } } peers { permanent_uuid: "8752865be9ad439ebb52d162fd73c205" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39067 } }
I20260926 20:56:58.426661 27390 raft_consensus.cc:374] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:56:58.426489 27388 consensus_queue.cc:260] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 37307 } } peers { permanent_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37483 } } peers { permanent_uuid: "8752865be9ad439ebb52d162fd73c205" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39067 } }
I20260926 20:56:58.426756 27390 raft_consensus.cc:729] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06f1d5c93fe49298b0535fb0f1a4d20, State: Initialized, Role: FOLLOWER
I20260926 20:56:58.427392 27390 consensus_queue.cc:260] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 37307 } } peers { permanent_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37483 } } peers { permanent_uuid: "8752865be9ad439ebb52d162fd73c205" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39067 } }
I20260926 20:56:58.428323 27388 ts_tablet_manager.cc:1428] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260926 20:56:58.428435 27370 heartbeater.cc:503] Master 127.8.249.190:46363 was elected leader, sending a full tablet report...
I20260926 20:56:58.429029 27390 ts_tablet_manager.cc:1428] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260926 20:56:58.551107 27371 tablet.cc:2369] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:56:58.581274 27241 tablet.cc:2369] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:56:58.618258 27110 tablet.cc:2369] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:56:58.636245 27396 raft_consensus.cc:482] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:56:58.636580 27396 raft_consensus.cc:504] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 37307 } } peers { permanent_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37483 } } peers { permanent_uuid: "8752865be9ad439ebb52d162fd73c205" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39067 } }
I20260926 20:56:58.638373 27396 leader_election.cc:288] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c3e9ccbf0fd49d39c86fc44b2bbba10 (127.8.249.130:37307), 8752865be9ad439ebb52d162fd73c205 (127.8.249.129:39067)
I20260926 20:56:58.645171 27387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.639127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:43291 (local address 127.8.249.129:39067)
0926 20:56:58.639357 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:58.639363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:58.639383 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:56:58.639463 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:58.639470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:58.639577 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:58.639688 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:58.640242 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.641229 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.643161 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.644235 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.644364 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.644416 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:58.644742 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 20:56:58.644867 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:56:58.645226 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.638634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39067 (local address 127.8.249.131:43291)
0926 20:56:58.638982 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:58.639001 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:58.639146 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:58.639708 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:58.639712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:58.639758 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:58.640051 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:58.640063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.641362 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.641387 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:58.642996 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:58.643011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.644402 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.644409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:58.644579 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.644612 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 20:56:58.644703 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:56:58.644865 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"spinlock_wait_cycles":22400}
I20260926 20:56:58.645570 27384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.638870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:37795 (local address 127.8.249.130:37307)
0926 20:56:58.639085 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:58.639092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:58.639775 (+ 683us) server_negotiation.cc:407] Connection header received
0926 20:56:58.639970 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:58.639975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:58.640064 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:58.640167 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:56:58.640858 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.641739 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.643683 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.644591 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.644760 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.644823 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:58.645167 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 20:56:58.645310 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 20:56:58.645670 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.638780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:37307 (local address 127.8.249.131:37795)
0926 20:56:58.639665 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:58.639684 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:56:58.639801 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:56:58.640235 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:56:58.640239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:56:58.640265 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:56:58.640661 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:58.640673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.641896 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.641900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:58.643500 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:56:58.643514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.644777 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.644783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:56:58.644992 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.645030 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 20:56:58.645120 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:56:58.645336 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":34,"thread_start_us":384,"threads_started":1}
I20260926 20:56:58.646461 27064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d12766803234942827144de85e467da" candidate_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8752865be9ad439ebb52d162fd73c205" is_pre_election: true
I20260926 20:56:58.646667 27195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d12766803234942827144de85e467da" candidate_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10" is_pre_election: true
I20260926 20:56:58.647110 27064 raft_consensus.cc:2454] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f06f1d5c93fe49298b0535fb0f1a4d20 in term 0.
I20260926 20:56:58.647239 27195 raft_consensus.cc:2454] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f06f1d5c93fe49298b0535fb0f1a4d20 in term 0.
I20260926 20:56:58.647837 27258 leader_election.cc:302] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c3e9ccbf0fd49d39c86fc44b2bbba10, 8752865be9ad439ebb52d162fd73c205, f06f1d5c93fe49298b0535fb0f1a4d20; no voters:
I20260926 20:56:58.648283 27396 raft_consensus.cc:2781] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:56:58.648471 27396 raft_consensus.cc:482] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:56:58.648587 27396 raft_consensus.cc:3037] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:58.650748 27396 raft_consensus.cc:504] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 37307 } } peers { permanent_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37483 } } peers { permanent_uuid: "8752865be9ad439ebb52d162fd73c205" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39067 } }
I20260926 20:56:58.651412 27396 leader_election.cc:288] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [CANDIDATE]: Term 1 election: Requested vote from peers 2c3e9ccbf0fd49d39c86fc44b2bbba10 (127.8.249.130:37307), 8752865be9ad439ebb52d162fd73c205 (127.8.249.129:39067)
I20260926 20:56:58.651919 27195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d12766803234942827144de85e467da" candidate_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10"
I20260926 20:56:58.651975 27064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d12766803234942827144de85e467da" candidate_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8752865be9ad439ebb52d162fd73c205"
I20260926 20:56:58.652112 27195 raft_consensus.cc:3037] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:58.652223 27064 raft_consensus.cc:3037] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:56:58.654320 27195 raft_consensus.cc:2454] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f06f1d5c93fe49298b0535fb0f1a4d20 in term 1.
I20260926 20:56:58.654382 27064 raft_consensus.cc:2454] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f06f1d5c93fe49298b0535fb0f1a4d20 in term 1.
I20260926 20:56:58.654734 27260 leader_election.cc:302] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3e9ccbf0fd49d39c86fc44b2bbba10, f06f1d5c93fe49298b0535fb0f1a4d20; no voters:
I20260926 20:56:58.655174 27396 raft_consensus.cc:2781] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:56:58.655696 27396 raft_consensus.cc:686] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [term 1 LEADER]: Becoming Leader. State: Replica: f06f1d5c93fe49298b0535fb0f1a4d20, State: Running, Role: LEADER
I20260926 20:56:58.656213 27396 consensus_queue.cc:237] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 37307 } } peers { permanent_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37483 } } peers { permanent_uuid: "8752865be9ad439ebb52d162fd73c205" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39067 } }
I20260926 20:56:58.661037 26922 catalog_manager.cc:5495] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 reported cstate change: term changed from 0 to 1, leader changed from <none> to f06f1d5c93fe49298b0535fb0f1a4d20 (127.8.249.131). New cstate: current_term: 1 leader_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f06f1d5c93fe49298b0535fb0f1a4d20" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8752865be9ad439ebb52d162fd73c205" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39067 } health_report { overall_health: UNKNOWN } } }
I20260926 20:56:58.678174 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.672702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.8.249.131:37483)
0926 20:56:58.672942 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:58.672947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:58.673047 (+ 100us) server_negotiation.cc:407] Connection header received
0926 20:56:58.673283 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:58.673287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:58.673361 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:58.673531 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:56:58.673896 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.674621 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.676077 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.676503 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.676638 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.677132 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:56:58.677618 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:56:58.677787 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:58.677836 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 20:56:58.677925 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:56:58.686228 27195 raft_consensus.cc:1261] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10 [term 1 FOLLOWER]: Refusing update from remote peer f06f1d5c93fe49298b0535fb0f1a4d20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:58.686247 27064 raft_consensus.cc:1261] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205 [term 1 FOLLOWER]: Refusing update from remote peer f06f1d5c93fe49298b0535fb0f1a4d20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:56:58.687258 27398 consensus_queue.cc:1035] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3e9ccbf0fd49d39c86fc44b2bbba10" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:58.687613 27396 consensus_queue.cc:1035] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8752865be9ad439ebb52d162fd73c205" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:56:58.700628 27403 mvcc.cc:205] Tried to move back new op lower bound from 7333708671729004544 to 7333708671620186112. Current Snapshot: MvccSnapshot[applied={T|T < 7333708671729004544}]
I20260926 20:56:58.713260 26922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:58.713562 26922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:56:58.716898 26922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21574a097e4d4994a3f9ba5c39cf4c3e: Sending DeleteTablet for 3 replicas of tablet 2d12766803234942827144de85e467da
I20260926 20:56:58.718079 27305 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d12766803234942827144de85e467da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:58 UTC) from {username='slave'} at 127.0.0.1:45658
I20260926 20:56:58.718109 27175 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d12766803234942827144de85e467da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:58 UTC) from {username='slave'} at 127.0.0.1:33286
I20260926 20:56:58.718595 27044 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d12766803234942827144de85e467da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:56:58 UTC) from {username='slave'} at 127.0.0.1:43850
I20260926 20:56:58.724936 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 26982
I20260926 20:56:58.729665 27412 tablet_replica.cc:330] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10: stopping tablet replica
I20260926 20:56:58.730173 27412 raft_consensus.cc:2229] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:58.730957 27412 raft_consensus.cc:2258] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:58.731570 27413 tablet_replica.cc:330] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205: stopping tablet replica
I20260926 20:56:58.734063 27412 ts_tablet_manager.cc:1905] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:58.734604 27413 raft_consensus.cc:2229] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:56:58.735188 27413 raft_consensus.cc:2258] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:58.741276 27411 tablet_replica.cc:330] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20: stopping tablet replica
I20260926 20:56:58.741547 27413 ts_tablet_manager.cc:1905] T 2d12766803234942827144de85e467da P 8752865be9ad439ebb52d162fd73c205: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:58.741788 27387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.726749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.8.249.129:39067)
0926 20:56:58.727125 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:58.727134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:58.727160 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:56:58.727245 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:58.727253 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:58.727385 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:58.727522 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:58.732671 (+ 5149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.734217 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.736929 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.737774 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.737901 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.739847 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:58.739862 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:58.739886 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:58.740863 (+ 977us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:58.740938 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:58.740965 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:58.741058 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:58.741286 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 20:56:58.741397 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260926 20:56:58.742540 27104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:58.745479 27412 ts_tablet_manager.cc:1918] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:58.745688 27412 log.cc:1196] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/2d12766803234942827144de85e467da
I20260926 20:56:58.746142 27412 ts_tablet_manager.cc:1939] T 2d12766803234942827144de85e467da P 2c3e9ccbf0fd49d39c86fc44b2bbba10: Deleting consensus metadata
I20260926 20:56:58.747335 27411 raft_consensus.cc:2229] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:56:58.747967 26909 catalog_manager.cc:4841] TS 2c3e9ccbf0fd49d39c86fc44b2bbba10 (127.8.249.130:37307): tablet 2d12766803234942827144de85e467da (table test-table [id=cc07e056a8b946a5a833773cce2d9695]) successfully deleted
I20260926 20:56:58.748340 27411 raft_consensus.cc:2258] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:56:58.751518 27411 ts_tablet_manager.cc:1905] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:56:58.759861 27411 ts_tablet_manager.cc:1918] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:56:58.760167 27411 log.cc:1196] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/2d12766803234942827144de85e467da
I20260926 20:56:58.760916 27411 ts_tablet_manager.cc:1939] T 2d12766803234942827144de85e467da P f06f1d5c93fe49298b0535fb0f1a4d20: Deleting consensus metadata
I20260926 20:56:58.763425 26910 catalog_manager.cc:4841] TS f06f1d5c93fe49298b0535fb0f1a4d20 (127.8.249.131:37483): tablet 2d12766803234942827144de85e467da (table test-table [id=cc07e056a8b946a5a833773cce2d9695]) successfully deleted
I20260926 20:56:58.843230 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 26982
W20260926 20:56:58.862258 26907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.249.129:39067 (error 108)
I20260926 20:56:58.862465 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 27113
I20260926 20:56:58.864243 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.863720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39067 (local address 127.0.0.1:43862)
0926 20:56:58.863909 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:58.864069 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 20:56:58.864512 26907 catalog_manager.cc:4584] TS 8752865be9ad439ebb52d162fd73c205 (127.8.249.129:39067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d12766803234942827144de85e467da: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39067: connect: Connection refused (error 111)
I20260926 20:56:58.869957 27384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.863668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.8.249.130:37307)
0926 20:56:58.863867 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:58.863875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:58.863893 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:56:58.863957 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:58.863961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:58.864045 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:58.864159 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:58.864615 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.865414 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.866662 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.867267 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.867395 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.868522 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:58.868554 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:58.868575 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:58.869152 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:58.869200 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:58.869218 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:58.869336 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:58.869463 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 20:56:58.869591 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:56:58.870682 27235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:58.918159 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.917600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39067 (local address 127.0.0.1:43870)
0926 20:56:58.917856 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:58.917972 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 20:56:58.919353 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.918953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39067 (local address 127.0.0.1:43878)
0926 20:56:58.919112 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:58.919202 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:56:58.961165 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 27113
I20260926 20:56:58.980203 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.979713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39067 (local address 127.0.0.1:43890)
0926 20:56:58.979953 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:58.980039 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 20:56:58.981261 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.980920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39067 (local address 127.0.0.1:43896)
0926 20:56:58.981068 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:58.981144 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 20:56:58.984086 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 27243
I20260926 20:56:58.991189 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:58.984873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45668 (local address 127.8.249.131:37483)
0926 20:56:58.985088 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 20:56:58.985094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:56:58.985112 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:56:58.985213 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:56:58.985218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:56:58.985309 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:56:58.985427 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:56:58.985830 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.986622 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.987973 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:56:58.988434 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:56:58.988574 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:56:58.989699 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:56:58.989707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:56:58.989721 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:56:58.990309 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:56:58.990352 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:56:58.990371 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:56:58.990509 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:56:58.990628 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 20:56:58.990723 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:56:58.991919 27365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:59.062053 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:59.061567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39067 (local address 127.0.0.1:43900)
0926 20:56:59.061759 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:59.061884 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 20:56:59.063246 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:59.062820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39067 (local address 127.0.0.1:43916)
0926 20:56:59.063028 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:59.063111 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260926 20:56:59.109388 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 27243
I20260926 20:56:59.128182 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 26892
I20260926 20:56:59.129329 26953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:56:59.241997 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:56:59.241373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39067 (local address 127.0.0.1:43932)
0926 20:56:59.241619 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 20:56:59.241736 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 20:56:59.242419 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 26892
2026-09-26T20:56:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2971 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:56:59 dist-test-slave-2dg3 krb5kdc[27422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:56:59 dist-test-slave-2dg3 krb5kdc[27422](info): set up 2 sockets
Sep 26 20:56:59 dist-test-slave-2dg3 krb5kdc[27422](info): commencing operation
krb5kdc: starting...
W20260926 20:57:00.363443 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.011s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:00 dist-test-slave-2dg3 krb5kdc[27422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:57:00.621949 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:35363
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:35363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:00.953301 27435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:00.953640 27435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:00.953730 27435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:00.960948 27435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:00.961062 27435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:00.961108 27435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:00.961166 27435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:00.961210 27435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:00.971278 27435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:35363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:35363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:00.972554 27435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:00.973776 27435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:00.985850 27443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:00.985845 27441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:00.986322 27440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:00.986591 27435 server_base.cc:1034] running on GCE node
I20260926 20:57:00.987418 27435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:00.988873 27435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:00.990104 27435 hybrid_clock.cc:648] HybridClock initialized: now 1790456220990052 us; error 61 us; skew 500 ppm
I20260926 20:57:00.990466 27435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:00 dist-test-slave-2dg3 krb5kdc[27422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456220, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:00.996615 27435 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:57:00.998054 27435 webserver.cc:460] Webserver started at http://127.8.249.190:43135/ using document root <none> and password file <none>
I20260926 20:57:00.998732 27435 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:00.998900 27435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:00.999212 27435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:01.002580 27435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "c758c93d06de40bbacf6cdb3af9fba2c"
format_stamp: "Formatted at 2026-09-26 20:57:00 on dist-test-slave-2dg3"
server_key: "5d3bb48b5397b97cadb990ac548a1090"
server_key_iv: "f194fa160d1394b572a3b6e54d4d10da"
server_key_version: "encryptionkey@0"
I20260926 20:57:01.003398 27435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "c758c93d06de40bbacf6cdb3af9fba2c"
format_stamp: "Formatted at 2026-09-26 20:57:00 on dist-test-slave-2dg3"
server_key: "5d3bb48b5397b97cadb990ac548a1090"
server_key_iv: "f194fa160d1394b572a3b6e54d4d10da"
server_key_version: "encryptionkey@0"
I20260926 20:57:01.008569 27435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:57:01.011942 27450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:01.013275 27435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:57:01.013417 27435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "c758c93d06de40bbacf6cdb3af9fba2c"
format_stamp: "Formatted at 2026-09-26 20:57:00 on dist-test-slave-2dg3"
server_key: "5d3bb48b5397b97cadb990ac548a1090"
server_key_iv: "f194fa160d1394b572a3b6e54d4d10da"
server_key_version: "encryptionkey@0"
I20260926 20:57:01.013578 27435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:01.043260 27435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:01.047364 27435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:01.047614 27435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:01.066737 27435 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:35363
I20260926 20:57:01.066757 27501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:35363 every 8 connection(s)
I20260926 20:57:01.068254 27435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:01.072149 27502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:01.077533 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 27435
I20260926 20:57:01.077713 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:01.078234 9190 external_mini_cluster.cc:1428] Setting key 77119ea179bd93568793ba867ea03aba
I20260926 20:57:01.083678 27502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c: Bootstrap starting.
I20260926 20:57:01.086906 27502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:01.087911 27502 log.cc:824] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:01.090969 27502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c: No bootstrap required, opened a new log
Sep 26 20:57:01 dist-test-slave-2dg3 krb5kdc[27422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:01.101449 27502 raft_consensus.cc:348] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c758c93d06de40bbacf6cdb3af9fba2c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 35363 } }
I20260926 20:57:01.101931 27502 raft_consensus.cc:374] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:01.102038 27502 raft_consensus.cc:729] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c758c93d06de40bbacf6cdb3af9fba2c, State: Initialized, Role: FOLLOWER
I20260926 20:57:01.102689 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:01.080305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43378 (local address 127.8.249.190:35363)
0926 20:57:01.081166 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:01.081178 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:01.081224 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:57:01.082994 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:01.083018 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:01.083485 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:01.083865 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:01.085416 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:01.086687 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:01.088180 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:01.089070 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:01.089264 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:01.093646 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:01.093657 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:01.093675 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:01.093718 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:01.097789 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:01.098556 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:01.098565 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:01.098574 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:01.098840 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:01.099226 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:01.099231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:01.099235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:01.100051 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:01.100347 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:01.100783 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 20:57:01.101262 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":368,"threads_started":1}
I20260926 20:57:01.102725 27502 consensus_queue.cc:260] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c758c93d06de40bbacf6cdb3af9fba2c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 35363 } }
I20260926 20:57:01.103102 27502 raft_consensus.cc:388] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:01.103257 27502 raft_consensus.cc:482] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:01.103449 27502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:01.106189 27502 raft_consensus.cc:504] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c758c93d06de40bbacf6cdb3af9fba2c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 35363 } }
I20260926 20:57:01.106616 27502 leader_election.cc:302] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c758c93d06de40bbacf6cdb3af9fba2c; no voters:
I20260926 20:57:01.107230 27502 leader_election.cc:288] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:01.107399 27507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:01.108184 27507 raft_consensus.cc:686] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [term 1 LEADER]: Becoming Leader. State: Replica: c758c93d06de40bbacf6cdb3af9fba2c, State: Running, Role: LEADER
I20260926 20:57:01.108719 27507 consensus_queue.cc:237] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c758c93d06de40bbacf6cdb3af9fba2c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 35363 } }
I20260926 20:57:01.109338 27502 sys_catalog.cc:564] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:01.111645 27509 sys_catalog.cc:455] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c758c93d06de40bbacf6cdb3af9fba2c. Latest consensus state: current_term: 1 leader_uuid: "c758c93d06de40bbacf6cdb3af9fba2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c758c93d06de40bbacf6cdb3af9fba2c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 35363 } } }
I20260926 20:57:01.112107 27509 sys_catalog.cc:458] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:01.112907 27508 sys_catalog.cc:455] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c758c93d06de40bbacf6cdb3af9fba2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c758c93d06de40bbacf6cdb3af9fba2c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 35363 } } }
I20260926 20:57:01.113335 27508 sys_catalog.cc:458] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:01.113610 27512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:01.120002 27512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:01.128857 27512 catalog_manager.cc:1347] Generated new cluster ID: 24f2af0eb44049af82f8ace55a8d3032
I20260926 20:57:01.128994 27512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:01.155187 27512 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:01.157194 27512 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:01.178365 27512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c: Generated new TSK 0
I20260926 20:57:01.179709 27512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:57:01.243818 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:35363
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:01.581988 27528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:01.582239 27528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:01.582352 27528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:01.589663 27528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:01.589764 27528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:01.589876 27528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:57:01.599504 27528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:01.600704 27528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:01.601962 27528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:01.616647 27534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:01.616721 27533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:01.616757 27536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:01.617712 27528 server_base.cc:1034] running on GCE node
I20260926 20:57:01.618485 27528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:01.619541 27528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:01.620795 27528 hybrid_clock.cc:648] HybridClock initialized: now 1790456221620740 us; error 53 us; skew 500 ppm
I20260926 20:57:01.621141 27528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:01 dist-test-slave-2dg3 krb5kdc[27422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456221, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:01.624342 27528 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:57:01.625773 27528 webserver.cc:460] Webserver started at http://127.8.249.129:41349/ using document root <none> and password file <none>
I20260926 20:57:01.626479 27528 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:01.626634 27528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:01.626987 27528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:01.629832 27528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "dbf88f1b8b8743ae9f383e365fb1747c"
format_stamp: "Formatted at 2026-09-26 20:57:01 on dist-test-slave-2dg3"
server_key: "c62a519e3e3b641ccf4085fd9e2bc2ba"
server_key_iv: "305d198daad2b47f5d790e033ea4ebd6"
server_key_version: "encryptionkey@0"
I20260926 20:57:01.630597 27528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "dbf88f1b8b8743ae9f383e365fb1747c"
format_stamp: "Formatted at 2026-09-26 20:57:01 on dist-test-slave-2dg3"
server_key: "c62a519e3e3b641ccf4085fd9e2bc2ba"
server_key_iv: "305d198daad2b47f5d790e033ea4ebd6"
server_key_version: "encryptionkey@0"
I20260926 20:57:01.635591 27528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 20:57:01.639067 27543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:01.640300 27528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:57:01.640487 27528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "dbf88f1b8b8743ae9f383e365fb1747c"
format_stamp: "Formatted at 2026-09-26 20:57:01 on dist-test-slave-2dg3"
server_key: "c62a519e3e3b641ccf4085fd9e2bc2ba"
server_key_iv: "305d198daad2b47f5d790e033ea4ebd6"
server_key_version: "encryptionkey@0"
I20260926 20:57:01.640683 27528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:01.675930 27528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:01.680613 27528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:01.680963 27528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:01.682005 27528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:01.683423 27528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:01.683525 27528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:01.683679 27528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:01.683753 27528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:01.717586 27528 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:39019
I20260926 20:57:01.717689 27655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:39019 every 8 connection(s)
I20260926 20:57:01.719051 27528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:57:01.719588 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 27528
I20260926 20:57:01.719787 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:57:01.720415 9190 external_mini_cluster.cc:1428] Setting key ec007bb414114e36e56aafd7b401e890
Sep 26 20:57:01 dist-test-slave-2dg3 krb5kdc[27422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456221, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:01.740661 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:01.721756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:39651 (local address 127.8.249.190:35363)
0926 20:57:01.722232 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:01.722238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:01.723562 (+ 1324us) server_negotiation.cc:407] Connection header received
0926 20:57:01.725499 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:01.725507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:01.725602 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:01.725726 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:01.727380 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:01.728582 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:01.730169 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:01.730639 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:01.730739 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:01.735031 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:01.735037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:01.735040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:01.735072 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:01.737585 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:01.738338 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:01.738343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:01.738345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:01.738474 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:01.738954 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:01.738958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:01.738960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:01.739219 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:01.739383 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:01.740085 (+ 702us) server_negotiation.cc:299] Negotiation successful
0926 20:57:01.740303 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":236,"threads_started":1}
I20260926 20:57:01.741704 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:01.722504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:35363 (local address 127.8.249.129:39651)
0926 20:57:01.723395 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:01.723442 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:01.725208 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:01.725932 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:01.725941 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:01.726498 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:01.727180 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:01.727199 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:01.728752 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:01.728755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:01.730009 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:01.730021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:01.730910 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:01.730918 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:01.731250 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:01.732018 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:01.732052 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:01.734740 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:01.737735 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:01.737743 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:01.737761 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:01.738193 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:01.738585 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:01.738589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:01.738593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:01.738778 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:01.739355 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:01.739363 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:01.739706 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 20:57:01.739973 (+ 267us) client_negotiation.cc:240] Negotiation successful
0926 20:57:01.740489 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":391,"threads_started":1}
I20260926 20:57:01.743449 27656 heartbeater.cc:347] Connected to a master server at 127.8.249.190:35363
I20260926 20:57:01.743759 27656 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:01.744549 27656 heartbeater.cc:511] Master 127.8.249.190:35363 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
I20260926 20:57:01.746681 27467 ts_manager.cc:194] Registered new tserver with Master: dbf88f1b8b8743ae9f383e365fb1747c (127.8.249.129:39019)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
I20260926 20:57:01.748445 27467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:39651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:57:01.774209 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:35363
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:02.103377 27662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:02.103703 27662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:02.103868 27662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:02.111274 27662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:02.111418 27662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:02.111595 27662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:57:02.121873 27662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:02.123255 27662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:02.124526 27662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:02.137269 27667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:02.137292 27670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:02.137530 27668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:02.138100 27662 server_base.cc:1034] running on GCE node
I20260926 20:57:02.138752 27662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:02.139753 27662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:02.140952 27662 hybrid_clock.cc:648] HybridClock initialized: now 1790456222140915 us; error 32 us; skew 500 ppm
I20260926 20:57:02.141258 27662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:02 dist-test-slave-2dg3 krb5kdc[27422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456222, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:02.144552 27662 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:57:02.146088 27662 webserver.cc:460] Webserver started at http://127.8.249.130:34177/ using document root <none> and password file <none>
I20260926 20:57:02.146839 27662 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:02.146983 27662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:02.147315 27662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:02.150233 27662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d"
format_stamp: "Formatted at 2026-09-26 20:57:02 on dist-test-slave-2dg3"
server_key: "e33f01d61834a7a74ec49ca3297316f8"
server_key_iv: "84bdaf28babb3a5dcc901bf70f481556"
server_key_version: "encryptionkey@0"
I20260926 20:57:02.151072 27662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d"
format_stamp: "Formatted at 2026-09-26 20:57:02 on dist-test-slave-2dg3"
server_key: "e33f01d61834a7a74ec49ca3297316f8"
server_key_iv: "84bdaf28babb3a5dcc901bf70f481556"
server_key_version: "encryptionkey@0"
I20260926 20:57:02.156225 27662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 20:57:02.159830 27677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:02.161203 27662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:57:02.161389 27662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d"
format_stamp: "Formatted at 2026-09-26 20:57:02 on dist-test-slave-2dg3"
server_key: "e33f01d61834a7a74ec49ca3297316f8"
server_key_iv: "84bdaf28babb3a5dcc901bf70f481556"
server_key_version: "encryptionkey@0"
I20260926 20:57:02.161615 27662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:02.191661 27662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:02.196033 27662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:02.196355 27662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:02.197391 27662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:02.198843 27662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:02.198956 27662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:02.199106 27662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:02.199193 27662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:02.232570 27662 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:34127
I20260926 20:57:02.232613 27789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:34127 every 8 connection(s)
I20260926 20:57:02.233865 27662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:57:02.239647 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 27662
I20260926 20:57:02.239835 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:57:02.240408 9190 external_mini_cluster.cc:1428] Setting key c9152bfc321e8d8d64eeb68903593cd2
Sep 26 20:57:02 dist-test-slave-2dg3 krb5kdc[27422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456222, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:02.255263 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.236460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:59325 (local address 127.8.249.190:35363)
0926 20:57:02.236665 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:02.236670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:02.237838 (+ 1168us) server_negotiation.cc:407] Connection header received
0926 20:57:02.240371 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:02.240376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:02.240453 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:02.240622 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:02.242423 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.243315 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.245088 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.245615 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.245765 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.249458 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:02.249463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:02.249467 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:02.249499 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:02.252220 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:02.252943 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:02.252949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:02.252952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:02.253106 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:02.253568 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:02.253573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:02.253576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:02.253852 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:02.254006 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:02.254669 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 20:57:02.254916 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 20:57:02.256321 27792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.236998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:35363 (local address 127.8.249.130:59325)
0926 20:57:02.237683 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:02.237725 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:02.240024 (+ 2299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:02.240868 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:02.240877 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:02.241555 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:02.242255 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.242275 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.243472 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.243480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.244925 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.244937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.245854 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.245861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.246105 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.246838 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:02.246872 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:02.249230 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:02.252367 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:02.252376 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:02.252393 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:02.252817 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:02.253239 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:02.253244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:02.253247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:02.253437 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:02.253980 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:02.253988 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:02.254331 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 20:57:02.254591 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 20:57:02.255147 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":281,"threads_started":1}
I20260926 20:57:02.257849 27790 heartbeater.cc:347] Connected to a master server at 127.8.249.190:35363
I20260926 20:57:02.258157 27790 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:02.258919 27790 heartbeater.cc:511] Master 127.8.249.190:35363 requested a full tablet report, sending...
I20260926 20:57:02.260391 27467 ts_manager.cc:194] Registered new tserver with Master: bc2159f51c8b41d2aa90c970b8bc5d5d (127.8.249.130:34127)
I20260926 20:57:02.261680 27467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:59325
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:57:02.294346 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:35363
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:02.620245 27795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:02.620519 27795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:02.620710 27795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:02.628041 27795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:02.628149 27795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:02.628355 27795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:57:02.638296 27795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:35363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:02.639703 27795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:02.641058 27795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:02.654327 27800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:02.654356 27801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:02.654392 27803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:02.655539 27795 server_base.cc:1034] running on GCE node
I20260926 20:57:02.656165 27795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:02.657212 27795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:02.658439 27795 hybrid_clock.cc:648] HybridClock initialized: now 1790456222658399 us; error 36 us; skew 500 ppm
I20260926 20:57:02.658780 27795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:02 dist-test-slave-2dg3 krb5kdc[27422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456222, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:02.661991 27795 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:57:02.663476 27795 webserver.cc:460] Webserver started at http://127.8.249.131:44659/ using document root <none> and password file <none>
I20260926 20:57:02.664111 27795 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:02.664208 27795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:02.664467 27795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:02.667462 27795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "43bff37e312047f384109cbeebf999de"
format_stamp: "Formatted at 2026-09-26 20:57:02 on dist-test-slave-2dg3"
server_key: "d1bd362b98aaefcac306991f017daa57"
server_key_iv: "9cfa3151feff1f11eb666338a89741e0"
server_key_version: "encryptionkey@0"
I20260926 20:57:02.668200 27795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "43bff37e312047f384109cbeebf999de"
format_stamp: "Formatted at 2026-09-26 20:57:02 on dist-test-slave-2dg3"
server_key: "d1bd362b98aaefcac306991f017daa57"
server_key_iv: "9cfa3151feff1f11eb666338a89741e0"
server_key_version: "encryptionkey@0"
I20260926 20:57:02.673136 27795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 20:57:02.676426 27810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:02.677750 27795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:57:02.677906 27795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "43bff37e312047f384109cbeebf999de"
format_stamp: "Formatted at 2026-09-26 20:57:02 on dist-test-slave-2dg3"
server_key: "d1bd362b98aaefcac306991f017daa57"
server_key_iv: "9cfa3151feff1f11eb666338a89741e0"
server_key_version: "encryptionkey@0"
I20260926 20:57:02.678118 27795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:02.691953 27795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:02.695894 27795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:02.696201 27795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:02.697180 27795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:02.698531 27795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:02.698617 27795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:02.698767 27795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:02.698846 27795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 20:57:02.732836 27795 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:40579
I20260926 20:57:02.732853 27922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:40579 every 8 connection(s)
I20260926 20:57:02.734193 27795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:57:02.738957 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 27795
I20260926 20:57:02.739185 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:57:02.739753 9190 external_mini_cluster.cc:1428] Setting key fb971c01b280c5e0e92cb3352b57807d
Sep 26 20:57:02 dist-test-slave-2dg3 krb5kdc[27422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456222, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:02.751148 27656 heartbeater.cc:503] Master 127.8.249.190:35363 was elected leader, sending a full tablet report...
I20260926 20:57:02.755385 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.736939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:57375 (local address 127.8.249.190:35363)
0926 20:57:02.737118 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:02.737122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:02.738232 (+ 1110us) server_negotiation.cc:407] Connection header received
0926 20:57:02.740824 (+ 2592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:02.740829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:02.740942 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:02.741049 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:02.742916 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.743750 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.745382 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.745798 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.745913 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.749129 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:02.749134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:02.749137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:02.749169 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:02.752110 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:02.752977 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:02.752981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:02.752983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:02.753108 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:02.753548 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:02.753553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:02.753556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:02.753840 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:02.754010 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:02.754816 (+ 806us) server_negotiation.cc:299] Negotiation successful
0926 20:57:02.755036 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:57:02.756361 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.737361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:35363 (local address 127.8.249.131:57375)
0926 20:57:02.738062 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:02.738106 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:02.740537 (+ 2431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:02.741327 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:02.741340 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:02.742041 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:02.742729 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.742754 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.743930 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.743934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.745229 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.745240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.745986 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.745991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.746180 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.746877 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:02.746899 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:02.748943 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:02.752255 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:02.752264 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:02.752281 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:02.752820 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:02.753215 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:02.753220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:02.753224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:02.753426 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:02.753973 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:02.753981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:02.754355 (+ 374us) client_negotiation.cc:769] Sending connection context
0926 20:57:02.754671 (+ 316us) client_negotiation.cc:240] Negotiation successful
0926 20:57:02.755232 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":265,"threads_started":1}
I20260926 20:57:02.757718 27923 heartbeater.cc:347] Connected to a master server at 127.8.249.190:35363
I20260926 20:57:02.758046 27923 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:02.758751 27923 heartbeater.cc:511] Master 127.8.249.190:35363 requested a full tablet report, sending...
I20260926 20:57:02.760090 27467 ts_manager.cc:194] Registered new tserver with Master: 43bff37e312047f384109cbeebf999de (127.8.249.131:40579)
I20260926 20:57:02.760820 27467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:57375
I20260926 20:57:02.767377 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:57:02.783890 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.771174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.8.249.190:35363)
0926 20:57:02.771323 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:02.771327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:02.771838 (+ 511us) server_negotiation.cc:407] Connection header received
0926 20:57:02.771996 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:02.772000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:02.772052 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:02.772139 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:02.773426 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.774112 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.775457 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.775852 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.775958 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.778509 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:02.778514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:02.778518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:02.778544 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:02.780888 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:02.781640 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:02.781644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:02.781647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:02.781819 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:02.782274 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:02.782277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:02.782279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:02.782620 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:02.782925 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:02.783387 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 20:57:02.783588 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:57:02.787786 27467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:57:02.794533 27467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:57:02.818456 27938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.807111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.8.249.130:34127)
0926 20:57:02.810390 (+ 3279us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:02.810399 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:02.810429 (+ 30us) server_negotiation.cc:407] Connection header received
0926 20:57:02.810528 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:02.810535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:02.810777 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:02.811014 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:02.811966 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.813432 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.815512 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.817196 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.817363 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.817606 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:02.817974 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 20:57:02.818154 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3130,"thread_start_us":3026,"threads_started":1}
I20260926 20:57:02.818519 27936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.806827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:34127 (local address 127.0.0.1:53734)
0926 20:57:02.809422 (+ 2595us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:02.809440 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:02.810389 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:02.811281 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:02.811289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:02.811315 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:02.811777 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.811791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.813559 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.813567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.815308 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.815327 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.817458 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.817466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.817670 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.817719 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 20:57:02.817922 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 20:57:02.818141 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2424,"thread_start_us":3667,"threads_started":1}
I20260926 20:57:02.820329 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.806529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39019 (local address 127.0.0.1:45314)
0926 20:57:02.807520 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:02.807582 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:02.807819 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:02.812059 (+ 4240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:02.812066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:02.812097 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:02.812961 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.812978 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.814894 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.814899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.817260 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.817277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.819438 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.819443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.819613 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.819664 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 20:57:02.819755 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:57:02.819894 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":59,"thread_start_us":618,"threads_started":1}
I20260926 20:57:02.820570 27937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.806993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.8.249.129:39019)
0926 20:57:02.810045 (+ 3052us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:02.810054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:02.810085 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:57:02.810206 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:02.810213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:02.810507 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:02.810811 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:02.813151 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.814707 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.817450 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.818980 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.819124 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.819281 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:02.820058 (+ 777us) server_negotiation.cc:299] Negotiation successful
0926 20:57:02.820224 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2863,"thread_start_us":2694,"threads_started":1}
I20260926 20:57:02.820866 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.810202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.8.249.131:40579)
0926 20:57:02.811866 (+ 1664us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:02.811876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:02.811908 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:57:02.812003 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:02.812009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:02.812207 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:02.812407 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:02.813060 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.814722 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.817409 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.818902 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.819032 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.819254 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:02.820334 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0926 20:57:02.820518 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1496,"thread_start_us":1335,"threads_started":1}
I20260926 20:57:02.821121 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.809512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:40579 (local address 127.0.0.1:57464)
0926 20:57:02.811159 (+ 1647us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:02.811177 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:02.811281 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:02.812409 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:02.812413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:02.812434 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:02.812870 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.812880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.814939 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.814946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.817181 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.817201 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.819416 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.819426 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.819607 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.819651 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:57:02.819739 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 20:57:02.819884 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1491,"mutex_wait_us":41,"thread_start_us":1345,"threads_started":1}
I20260926 20:57:02.823413 27725 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3a5f11d73e4427a5ab31fb791fc77b (DEFAULT_TABLE table=test-table [id=a476ea8489bc4b0585d09a030b97cd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:02.824272 27591 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3a5f11d73e4427a5ab31fb791fc77b (DEFAULT_TABLE table=test-table [id=a476ea8489bc4b0585d09a030b97cd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:02.824954 27725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3a5f11d73e4427a5ab31fb791fc77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:02.824754 27858 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3a5f11d73e4427a5ab31fb791fc77b (DEFAULT_TABLE table=test-table [id=a476ea8489bc4b0585d09a030b97cd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:02.825443 27591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3a5f11d73e4427a5ab31fb791fc77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:02.826059 27858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3a5f11d73e4427a5ab31fb791fc77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:02.836536 27941 tablet_bootstrap.cc:492] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c: Bootstrap starting.
I20260926 20:57:02.839699 27942 tablet_bootstrap.cc:492] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d: Bootstrap starting.
I20260926 20:57:02.841774 27941 tablet_bootstrap.cc:654] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:02.843158 27941 log.cc:824] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:02.843293 27942 tablet_bootstrap.cc:654] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:02.844650 27942 log.cc:824] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:02.844749 27943 tablet_bootstrap.cc:492] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de: Bootstrap starting.
I20260926 20:57:02.846662 27941 tablet_bootstrap.cc:492] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c: No bootstrap required, opened a new log
I20260926 20:57:02.847003 27941 ts_tablet_manager.cc:1397] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 20:57:02.848397 27942 tablet_bootstrap.cc:492] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d: No bootstrap required, opened a new log
I20260926 20:57:02.848464 27943 tablet_bootstrap.cc:654] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:02.848690 27942 ts_tablet_manager.cc:1397] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 20:57:02.849745 27943 log.cc:824] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:02.853570 27943 tablet_bootstrap.cc:492] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de: No bootstrap required, opened a new log
I20260926 20:57:02.853875 27943 ts_tablet_manager.cc:1397] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260926 20:57:02.855603 27941 raft_consensus.cc:348] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf88f1b8b8743ae9f383e365fb1747c" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39019 } } peers { permanent_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 34127 } } peers { permanent_uuid: "43bff37e312047f384109cbeebf999de" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40579 } }
I20260926 20:57:02.856014 27941 raft_consensus.cc:374] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:02.856143 27941 raft_consensus.cc:729] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf88f1b8b8743ae9f383e365fb1747c, State: Initialized, Role: FOLLOWER
I20260926 20:57:02.856798 27941 consensus_queue.cc:260] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf88f1b8b8743ae9f383e365fb1747c" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39019 } } peers { permanent_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 34127 } } peers { permanent_uuid: "43bff37e312047f384109cbeebf999de" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40579 } }
I20260926 20:57:02.858682 27941 ts_tablet_manager.cc:1428] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260926 20:57:02.859349 27942 raft_consensus.cc:348] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf88f1b8b8743ae9f383e365fb1747c" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39019 } } peers { permanent_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 34127 } } peers { permanent_uuid: "43bff37e312047f384109cbeebf999de" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40579 } }
I20260926 20:57:02.859807 27942 raft_consensus.cc:374] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:02.859941 27942 raft_consensus.cc:729] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2159f51c8b41d2aa90c970b8bc5d5d, State: Initialized, Role: FOLLOWER
I20260926 20:57:02.860677 27942 consensus_queue.cc:260] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf88f1b8b8743ae9f383e365fb1747c" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39019 } } peers { permanent_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 34127 } } peers { permanent_uuid: "43bff37e312047f384109cbeebf999de" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40579 } }
I20260926 20:57:02.863531 27790 heartbeater.cc:503] Master 127.8.249.190:35363 was elected leader, sending a full tablet report...
I20260926 20:57:02.864217 27942 ts_tablet_manager.cc:1428] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260926 20:57:02.865059 27943 raft_consensus.cc:348] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf88f1b8b8743ae9f383e365fb1747c" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39019 } } peers { permanent_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 34127 } } peers { permanent_uuid: "43bff37e312047f384109cbeebf999de" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40579 } }
I20260926 20:57:02.865440 27943 raft_consensus.cc:374] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:02.865542 27943 raft_consensus.cc:729] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bff37e312047f384109cbeebf999de, State: Initialized, Role: FOLLOWER
I20260926 20:57:02.866189 27943 consensus_queue.cc:260] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf88f1b8b8743ae9f383e365fb1747c" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39019 } } peers { permanent_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 34127 } } peers { permanent_uuid: "43bff37e312047f384109cbeebf999de" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40579 } }
I20260926 20:57:02.867322 27923 heartbeater.cc:503] Master 127.8.249.190:35363 was elected leader, sending a full tablet report...
I20260926 20:57:02.867908 27943 ts_tablet_manager.cc:1428] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260926 20:57:02.924295 27948 raft_consensus.cc:482] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:57:02.924589 27948 raft_consensus.cc:504] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf88f1b8b8743ae9f383e365fb1747c" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39019 } } peers { permanent_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 34127 } } peers { permanent_uuid: "43bff37e312047f384109cbeebf999de" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40579 } }
I20260926 20:57:02.926229 27948 leader_election.cc:288] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbf88f1b8b8743ae9f383e365fb1747c (127.8.249.129:39019), 43bff37e312047f384109cbeebf999de (127.8.249.131:40579)
I20260926 20:57:02.932997 27937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.926717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:39133 (local address 127.8.249.129:39019)
0926 20:57:02.927012 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:02.927022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:02.927819 (+ 797us) server_negotiation.cc:407] Connection header received
0926 20:57:02.928042 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:02.928046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:02.928105 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:02.928200 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:02.928783 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.929512 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.931296 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.932074 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.932253 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.932324 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:02.932608 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 20:57:02.932732 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 20:57:02.933105 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.926783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:41727 (local address 127.8.249.131:40579)
0926 20:57:02.927048 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:02.927057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:02.927908 (+ 851us) server_negotiation.cc:407] Connection header received
0926 20:57:02.928110 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:02.928117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:02.928181 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:02.928278 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:02.928932 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.929679 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.931467 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.932189 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.932371 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.932453 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:02.932742 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 20:57:02.932860 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:57:02.933223 27951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.926702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:40579 (local address 127.8.249.130:41727)
0926 20:57:02.927794 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:02.927808 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:02.927970 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:02.928297 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:02.928301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:02.928380 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:02.928765 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.928773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.929819 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.929829 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.931278 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.931288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.932402 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.932409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.932592 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.932613 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:57:02.932708 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:57:02.932824 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"spinlock_wait_cycles":23424,"thread_start_us":567,"threads_started":1}
I20260926 20:57:02.933621 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.926423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:39019 (local address 127.8.249.130:39133)
0926 20:57:02.927699 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:02.927715 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:02.927895 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:02.928214 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:02.928218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:02.928242 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:02.928638 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.928647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.929633 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.929641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.931145 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:02.931155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.932275 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.932281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:02.932450 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.932475 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:57:02.932584 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 20:57:02.933027 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":1,"thread_start_us":837,"threads_started":1}
I20260926 20:57:02.934386 27878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3a5f11d73e4427a5ab31fb791fc77b" candidate_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bff37e312047f384109cbeebf999de" is_pre_election: true
I20260926 20:57:02.934701 27611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3a5f11d73e4427a5ab31fb791fc77b" candidate_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbf88f1b8b8743ae9f383e365fb1747c" is_pre_election: true
I20260926 20:57:02.934964 27878 raft_consensus.cc:2454] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc2159f51c8b41d2aa90c970b8bc5d5d in term 0.
I20260926 20:57:02.935281 27611 raft_consensus.cc:2454] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc2159f51c8b41d2aa90c970b8bc5d5d in term 0.
I20260926 20:57:02.935631 27681 leader_election.cc:302] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bff37e312047f384109cbeebf999de, bc2159f51c8b41d2aa90c970b8bc5d5d; no voters:
I20260926 20:57:02.935987 27948 raft_consensus.cc:2781] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:57:02.936127 27948 raft_consensus.cc:482] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:57:02.936225 27948 raft_consensus.cc:3037] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:02.938326 27948 raft_consensus.cc:504] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf88f1b8b8743ae9f383e365fb1747c" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39019 } } peers { permanent_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 34127 } } peers { permanent_uuid: "43bff37e312047f384109cbeebf999de" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40579 } }
I20260926 20:57:02.938961 27948 leader_election.cc:288] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [CANDIDATE]: Term 1 election: Requested vote from peers dbf88f1b8b8743ae9f383e365fb1747c (127.8.249.129:39019), 43bff37e312047f384109cbeebf999de (127.8.249.131:40579)
I20260926 20:57:02.939343 27611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3a5f11d73e4427a5ab31fb791fc77b" candidate_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbf88f1b8b8743ae9f383e365fb1747c"
I20260926 20:57:02.939411 27878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3a5f11d73e4427a5ab31fb791fc77b" candidate_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bff37e312047f384109cbeebf999de"
I20260926 20:57:02.939551 27611 raft_consensus.cc:3037] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:02.939585 27878 raft_consensus.cc:3037] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:02.942646 27611 raft_consensus.cc:2454] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc2159f51c8b41d2aa90c970b8bc5d5d in term 1.
I20260926 20:57:02.942646 27878 raft_consensus.cc:2454] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc2159f51c8b41d2aa90c970b8bc5d5d in term 1.
I20260926 20:57:02.943081 27679 leader_election.cc:302] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc2159f51c8b41d2aa90c970b8bc5d5d, dbf88f1b8b8743ae9f383e365fb1747c; no voters:
I20260926 20:57:02.943430 27948 raft_consensus.cc:2781] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:02.944041 27948 raft_consensus.cc:686] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [term 1 LEADER]: Becoming Leader. State: Replica: bc2159f51c8b41d2aa90c970b8bc5d5d, State: Running, Role: LEADER
I20260926 20:57:02.944612 27948 consensus_queue.cc:237] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf88f1b8b8743ae9f383e365fb1747c" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39019 } } peers { permanent_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 34127 } } peers { permanent_uuid: "43bff37e312047f384109cbeebf999de" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40579 } }
I20260926 20:57:02.949124 27467 catalog_manager.cc:5495] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d reported cstate change: term changed from 0 to 1, leader changed from <none> to bc2159f51c8b41d2aa90c970b8bc5d5d (127.8.249.130). New cstate: current_term: 1 leader_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf88f1b8b8743ae9f383e365fb1747c" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc2159f51c8b41d2aa90c970b8bc5d5d" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 34127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43bff37e312047f384109cbeebf999de" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40579 } health_report { overall_health: UNKNOWN } } }
W20260926 20:57:02.971616 27657 tablet.cc:2369] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:57:02.976889 27938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:02.971612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.8.249.130:34127)
0926 20:57:02.971839 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:02.971848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:02.971876 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:57:02.971959 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:02.971962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:02.972036 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:02.972162 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:02.972635 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.973319 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.974672 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:02.975139 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:02.975246 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:02.975739 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:02.976245 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:02.976378 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:02.976509 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 20:57:02.976577 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:57:02.985291 27611 raft_consensus.cc:1261] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c [term 1 FOLLOWER]: Refusing update from remote peer bc2159f51c8b41d2aa90c970b8bc5d5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:57:02.985313 27878 raft_consensus.cc:1261] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de [term 1 FOLLOWER]: Refusing update from remote peer bc2159f51c8b41d2aa90c970b8bc5d5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 20:57:02.985975 27924 tablet.cc:2369] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:57:02.985984 27791 tablet.cc:2369] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:57:02.986438 27952 consensus_queue.cc:1035] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bff37e312047f384109cbeebf999de" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:57:02.986852 27948 consensus_queue.cc:1035] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbf88f1b8b8743ae9f383e365fb1747c" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 39019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:57:03.011700 27466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:57:03.011986 27466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:57:03.015597 27466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c758c93d06de40bbacf6cdb3af9fba2c: Sending DeleteTablet for 3 replicas of tablet 2d3a5f11d73e4427a5ab31fb791fc77b
I20260926 20:57:03.016803 27725 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3a5f11d73e4427a5ab31fb791fc77b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:03 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:53734
I20260926 20:57:03.016848 27591 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3a5f11d73e4427a5ab31fb791fc77b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:03 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:45314
I20260926 20:57:03.016855 27858 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3a5f11d73e4427a5ab31fb791fc77b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:03 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:57464
I20260926 20:57:03.021813 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 27528
I20260926 20:57:03.032990 27965 tablet_replica.cc:330] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d: stopping tablet replica
I20260926 20:57:03.033583 27965 raft_consensus.cc:2229] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:57:03.034490 27965 raft_consensus.cc:2258] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:57:03.035652 27966 tablet_replica.cc:330] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c: stopping tablet replica
I20260926 20:57:03.036182 27966 raft_consensus.cc:2229] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:57:03.036846 27967 tablet_replica.cc:330] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de: stopping tablet replica
I20260926 20:57:03.037243 27967 raft_consensus.cc:2229] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:57:03.037766 27967 raft_consensus.cc:2258] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:57:03.037866 27965 ts_tablet_manager.cc:1905] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:03.038508 27966 raft_consensus.cc:2258] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:57:03.039669 27967 ts_tablet_manager.cc:1905] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:03.040783 27966 ts_tablet_manager.cc:1905] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:03.046322 27967 ts_tablet_manager.cc:1918] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:57:03.046548 27967 log.cc:1196] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/2d3a5f11d73e4427a5ab31fb791fc77b
I20260926 20:57:03.047134 27967 ts_tablet_manager.cc:1939] T 2d3a5f11d73e4427a5ab31fb791fc77b P 43bff37e312047f384109cbeebf999de: Deleting consensus metadata
I20260926 20:57:03.048573 27966 ts_tablet_manager.cc:1918] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:57:03.048794 27966 log.cc:1196] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/2d3a5f11d73e4427a5ab31fb791fc77b
I20260926 20:57:03.049154 27454 catalog_manager.cc:4841] TS 43bff37e312047f384109cbeebf999de (127.8.249.131:40579): tablet 2d3a5f11d73e4427a5ab31fb791fc77b (table test-table [id=a476ea8489bc4b0585d09a030b97cd0c]) successfully deleted
I20260926 20:57:03.049460 27966 ts_tablet_manager.cc:1939] T 2d3a5f11d73e4427a5ab31fb791fc77b P dbf88f1b8b8743ae9f383e365fb1747c: Deleting consensus metadata
Sep 26 20:57:03 dist-test-slave-2dg3 krb5kdc[27422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:57:03.051285 27452 catalog_manager.cc:4841] TS dbf88f1b8b8743ae9f383e365fb1747c (127.8.249.129:39019): tablet 2d3a5f11d73e4427a5ab31fb791fc77b (table test-table [id=a476ea8489bc4b0585d09a030b97cd0c]) successfully deleted
I20260926 20:57:03.052269 27965 ts_tablet_manager.cc:1918] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:57:03.052451 27965 log.cc:1196] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/2d3a5f11d73e4427a5ab31fb791fc77b
I20260926 20:57:03.052886 27965 ts_tablet_manager.cc:1939] T 2d3a5f11d73e4427a5ab31fb791fc77b P bc2159f51c8b41d2aa90c970b8bc5d5d: Deleting consensus metadata
I20260926 20:57:03.054327 27453 catalog_manager.cc:4841] TS bc2159f51c8b41d2aa90c970b8bc5d5d (127.8.249.130:34127): tablet 2d3a5f11d73e4427a5ab31fb791fc77b (table test-table [id=a476ea8489bc4b0585d09a030b97cd0c]) successfully deleted
I20260926 20:57:03.057928 27937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:03.023633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.8.249.129:39019)
0926 20:57:03.023874 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:03.023884 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:03.040506 (+ 16622us) server_negotiation.cc:407] Connection header received
0926 20:57:03.040716 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:03.040722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:03.040822 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:03.040947 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:03.042714 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:03.043882 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:03.045956 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:03.046641 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:03.046761 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:03.051939 (+ 5178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:03.051950 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:03.051963 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:03.051997 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:03.055098 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:03.055835 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:03.055842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:03.055845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:03.055988 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:03.056372 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:03.056379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:03.056382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:03.056718 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:03.057031 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:03.057333 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 20:57:03.057547 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:57:03.058599 27651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:03.166397 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 27528
I20260926 20:57:03.185055 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 27662
Sep 26 20:57:03 dist-test-slave-2dg3 krb5kdc[27422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:57:03.202531 27938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:03.185919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.8.249.130:34127)
0926 20:57:03.186114 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:03.186119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:03.186136 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:57:03.186249 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:03.186253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:03.186348 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:03.186432 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:03.188131 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:03.188912 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:03.191000 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:03.191414 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:03.191513 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:03.196375 (+ 4862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:03.196388 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:03.196401 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:03.196438 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:03.199694 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:03.200379 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:03.200386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:03.200389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:03.200523 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:03.200991 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:03.200994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:03.200996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:03.201361 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:03.201666 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:03.201963 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 20:57:03.202182 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:57:03.203312 27785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:03.317762 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 27662
I20260926 20:57:03.336318 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 27795
Sep 26 20:57:03 dist-test-slave-2dg3 krb5kdc[27422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:57:03.350988 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:03.337069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.8.249.131:40579)
0926 20:57:03.337228 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:03.337232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:03.337248 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:57:03.337414 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:03.337419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:03.337485 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:03.337570 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:03.339075 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:03.339784 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:03.340990 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:03.341360 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:03.341449 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:03.345747 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:03.345762 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:03.345779 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:03.345815 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:03.348685 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:03.349307 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:03.349313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:03.349316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:03.349438 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:03.349745 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:03.349748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:03.349750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:03.350083 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:03.350289 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:03.350505 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 20:57:03.350662 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:57:03.351615 27918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:03.440780 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 27795
I20260926 20:57:03.457680 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 27435
I20260926 20:57:03.458766 27497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:03.569928 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 27435
2026-09-26T20:57:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4328 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T20:57:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:03Z Disabled control of system clock
I20260926 20:57:03.648192 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36295
--webserver_interface=127.8.249.190
--webserver_port=0
--builtin_ntp_servers=127.8.249.148:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:36295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:57:03.974092 27976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:03.974390 27976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:03.974534 27976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:03.981971 27976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:03.982084 27976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:03.982161 27976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:03.982249 27976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:03.982313 27976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:03.991707 27976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:36295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:03.993045 27976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:03.994954 27976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:04.007048 27982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:04.007134 27984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:04.007087 27981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:04.008153 27976 server_base.cc:1034] running on GCE node
I20260926 20:57:04.009011 27976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:04.010473 27976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:04.011669 27976 hybrid_clock.cc:648] HybridClock initialized: now 1790456224011631 us; error 28 us; skew 500 ppm
I20260926 20:57:04.012058 27976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:57:04.014256 27976 webserver.cc:460] Webserver started at http://127.8.249.190:43523/ using document root <none> and password file <none>
I20260926 20:57:04.014961 27976 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:04.015053 27976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:04.015321 27976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:04.018128 27976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "0b5af162d09e4f3ca882bca09ed64257"
format_stamp: "Formatted at 2026-09-26 20:57:04 on dist-test-slave-2dg3"
server_key: "fd201fa473d557d69ad5cdf2bb003d79"
server_key_iv: "d7f47100e6e96fc29e62cc6b46edd3a8"
server_key_version: "encryptionkey@0"
I20260926 20:57:04.018919 27976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "0b5af162d09e4f3ca882bca09ed64257"
format_stamp: "Formatted at 2026-09-26 20:57:04 on dist-test-slave-2dg3"
server_key: "fd201fa473d557d69ad5cdf2bb003d79"
server_key_iv: "d7f47100e6e96fc29e62cc6b46edd3a8"
server_key_version: "encryptionkey@0"
I20260926 20:57:04.023989 27976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:57:04.027590 27990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:04.029007 27976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:57:04.029212 27976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "0b5af162d09e4f3ca882bca09ed64257"
format_stamp: "Formatted at 2026-09-26 20:57:04 on dist-test-slave-2dg3"
server_key: "fd201fa473d557d69ad5cdf2bb003d79"
server_key_iv: "d7f47100e6e96fc29e62cc6b46edd3a8"
server_key_version: "encryptionkey@0"
I20260926 20:57:04.029440 27976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:04.048182 27976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:04.049124 27976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:04.049432 27976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:04.068140 27976 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:36295
I20260926 20:57:04.068171 28041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:36295 every 8 connection(s)
I20260926 20:57:04.069593 27976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:04.071143 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 27976
I20260926 20:57:04.071317 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:04.071830 9190 external_mini_cluster.cc:1428] Setting key d70a358e59ff7dfcb0ffe7d8912a1753
I20260926 20:57:04.074340 28042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:04.085129 28045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:04.073689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.8.249.190:36295)
0926 20:57:04.074701 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:04.074712 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:04.074752 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:57:04.076388 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:04.076411 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:04.076864 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:04.077284 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:04.077849 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:04.079252 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:04.080620 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:04.081284 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:04.081498 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:04.082742 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:04.082749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:04.082766 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:04.083431 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:04.083507 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:04.083528 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:04.083656 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:04.083887 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 20:57:04.084185 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":331,"threads_started":1}
I20260926 20:57:04.085377 28042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257: Bootstrap starting.
I20260926 20:57:04.089051 28042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:04.090040 28042 log.cc:824] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:04.092732 28042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257: No bootstrap required, opened a new log
I20260926 20:57:04.099875 28042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5af162d09e4f3ca882bca09ed64257" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36295 } }
I20260926 20:57:04.100348 28042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:04.100474 28042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5af162d09e4f3ca882bca09ed64257, State: Initialized, Role: FOLLOWER
I20260926 20:57:04.101069 28042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5af162d09e4f3ca882bca09ed64257" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36295 } }
I20260926 20:57:04.101264 28042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:04.101359 28042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:04.101487 28042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:04.103609 28042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5af162d09e4f3ca882bca09ed64257" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36295 } }
I20260926 20:57:04.104038 28042 leader_election.cc:302] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b5af162d09e4f3ca882bca09ed64257; no voters:
I20260926 20:57:04.104547 28042 leader_election.cc:288] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:04.104714 28047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:04.105450 28047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [term 1 LEADER]: Becoming Leader. State: Replica: 0b5af162d09e4f3ca882bca09ed64257, State: Running, Role: LEADER
I20260926 20:57:04.105962 28047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5af162d09e4f3ca882bca09ed64257" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36295 } }
I20260926 20:57:04.106739 28042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:04.109031 28048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b5af162d09e4f3ca882bca09ed64257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5af162d09e4f3ca882bca09ed64257" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36295 } } }
I20260926 20:57:04.109037 28049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b5af162d09e4f3ca882bca09ed64257. Latest consensus state: current_term: 1 leader_uuid: "0b5af162d09e4f3ca882bca09ed64257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5af162d09e4f3ca882bca09ed64257" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36295 } } }
I20260926 20:57:04.109624 28049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:04.109627 28048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:04.111511 28055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:04.116777 28055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:04.125821 28055 catalog_manager.cc:1347] Generated new cluster ID: b544157641be4a5c9294842e2506a267
I20260926 20:57:04.125953 28055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:04.137367 28055 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:04.139194 28055 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:04.154230 28055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257: Generated new TSK 0
I20260926 20:57:04.155474 28055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:04.159770 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36295
--builtin_ntp_servers=127.8.249.148:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:57:04.491505 28066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:04.491806 28066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:04.491892 28066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:04.499339 28066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:04.499460 28066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:04.499569 28066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:57:04.509400 28066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:04.510694 28066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:04.512703 28066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:04.525717 28071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:04.525650 28074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:04.525739 28072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:04.526530 28066 server_base.cc:1034] running on GCE node
I20260926 20:57:04.527230 28066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:04.528165 28066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:04.529361 28066 hybrid_clock.cc:648] HybridClock initialized: now 1790456224529326 us; error 38 us; skew 500 ppm
I20260926 20:57:04.529683 28066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:57:04.532021 28066 webserver.cc:460] Webserver started at http://127.8.249.129:37995/ using document root <none> and password file <none>
I20260926 20:57:04.532770 28066 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:04.532863 28066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:04.533219 28066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:04.536226 28066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "9523a2b5d6014f54bf8e89e420fd3aeb"
format_stamp: "Formatted at 2026-09-26 20:57:04 on dist-test-slave-2dg3"
server_key: "c36f634c5c0a3b5e0546a4fdcf330073"
server_key_iv: "c3fac7f038cad6f9a587dcd9f5342519"
server_key_version: "encryptionkey@0"
I20260926 20:57:04.537037 28066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "9523a2b5d6014f54bf8e89e420fd3aeb"
format_stamp: "Formatted at 2026-09-26 20:57:04 on dist-test-slave-2dg3"
server_key: "c36f634c5c0a3b5e0546a4fdcf330073"
server_key_iv: "c3fac7f038cad6f9a587dcd9f5342519"
server_key_version: "encryptionkey@0"
I20260926 20:57:04.542124 28066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 20:57:04.545627 28080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:04.547047 28066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 20:57:04.547220 28066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "9523a2b5d6014f54bf8e89e420fd3aeb"
format_stamp: "Formatted at 2026-09-26 20:57:04 on dist-test-slave-2dg3"
server_key: "c36f634c5c0a3b5e0546a4fdcf330073"
server_key_iv: "c3fac7f038cad6f9a587dcd9f5342519"
server_key_version: "encryptionkey@0"
I20260926 20:57:04.547439 28066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:04.567981 28066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:04.568832 28066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:04.569160 28066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:04.570145 28066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:04.571710 28066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:04.571843 28066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:04.571974 28066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:04.572062 28066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:04.605871 28066 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:41235
I20260926 20:57:04.605962 28192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:41235 every 8 connection(s)
I20260926 20:57:04.607342 28066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:57:04.614517 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 28066
I20260926 20:57:04.614697 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:57:04.615190 9190 external_mini_cluster.cc:1428] Setting key e9454966762011742f6c8ed7e5192a59
I20260926 20:57:04.617966 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36295
--builtin_ntp_servers=127.8.249.148:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:57:04.620566 28195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:04.610051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:35233 (local address 127.8.249.190:36295)
0926 20:57:04.610569 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:04.610575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:04.611780 (+ 1205us) server_negotiation.cc:407] Connection header received
0926 20:57:04.613532 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:04.613536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:04.613602 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:04.613694 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:04.614918 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:04.615673 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:04.617647 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:04.618064 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:04.618203 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:04.619676 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:04.619682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:04.619687 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:04.619726 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:04.619762 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:04.619773 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:04.619867 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:04.620126 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 20:57:04.620218 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":243,"threads_started":1}
I20260926 20:57:04.621496 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:04.610624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36295 (local address 127.8.249.129:35233)
0926 20:57:04.611653 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:04.611687 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:04.613300 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:04.613881 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:04.613888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:04.613967 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:04.614737 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:04.614764 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:04.615826 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:04.615830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:04.617492 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:04.617506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:04.618421 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:04.618426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:04.618618 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:04.619366 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:57:04.619368 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:04.619412 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:57:04.619486 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:04.619866 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:04.619877 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:04.619887 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:57:04.620085 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 20:57:04.620478 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":308,"threads_started":1}
I20260926 20:57:04.622973 28193 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36295
I20260926 20:57:04.623307 28193 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:04.624185 28193 heartbeater.cc:511] Master 127.8.249.190:36295 requested a full tablet report, sending...
I20260926 20:57:04.626466 28007 ts_manager.cc:194] Registered new tserver with Master: 9523a2b5d6014f54bf8e89e420fd3aeb (127.8.249.129:41235)
I20260926 20:57:04.627998 28007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.249.129:35233
W20260926 20:57:04.948472 28197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:04.948832 28197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:04.948985 28197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:04.956565 28197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:04.956705 28197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:04.956884 28197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:57:04.965893 28197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:04.967224 28197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:04.969041 28197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:04.982070 28205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:04.982118 28202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:04.982141 28203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:04.982947 28197 server_base.cc:1034] running on GCE node
I20260926 20:57:04.983546 28197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:04.984478 28197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:04.985711 28197 hybrid_clock.cc:648] HybridClock initialized: now 1790456224985667 us; error 40 us; skew 500 ppm
I20260926 20:57:04.986032 28197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:57:04.988319 28197 webserver.cc:460] Webserver started at http://127.8.249.130:38685/ using document root <none> and password file <none>
I20260926 20:57:04.989039 28197 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:04.989172 28197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:04.989497 28197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:04.992517 28197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "9075b946e81c497f85026e7134e66eb5"
format_stamp: "Formatted at 2026-09-26 20:57:04 on dist-test-slave-2dg3"
server_key: "9b49e328e33a5d32ce250ead45089aba"
server_key_iv: "2161b04dc5353ebfde5bbb3b4d2f2524"
server_key_version: "encryptionkey@0"
I20260926 20:57:04.993359 28197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "9075b946e81c497f85026e7134e66eb5"
format_stamp: "Formatted at 2026-09-26 20:57:04 on dist-test-slave-2dg3"
server_key: "9b49e328e33a5d32ce250ead45089aba"
server_key_iv: "2161b04dc5353ebfde5bbb3b4d2f2524"
server_key_version: "encryptionkey@0"
I20260926 20:57:04.998354 28197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 20:57:05.001741 28211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:05.003027 28197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:05.003190 28197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "9075b946e81c497f85026e7134e66eb5"
format_stamp: "Formatted at 2026-09-26 20:57:04 on dist-test-slave-2dg3"
server_key: "9b49e328e33a5d32ce250ead45089aba"
server_key_iv: "2161b04dc5353ebfde5bbb3b4d2f2524"
server_key_version: "encryptionkey@0"
I20260926 20:57:05.003405 28197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:05.027395 28197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:05.028218 28197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:05.028527 28197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:05.029500 28197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:05.030982 28197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:05.031100 28197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:05.031229 28197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:05.031324 28197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:05.064224 28197 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:32821
I20260926 20:57:05.064302 28323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:32821 every 8 connection(s)
I20260926 20:57:05.065557 28197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:57:05.075151 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 28197
I20260926 20:57:05.075349 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:57:05.075873 9190 external_mini_cluster.cc:1428] Setting key b163c902c9107718e40f24876f22b090
I20260926 20:57:05.079463 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36295
--builtin_ntp_servers=127.8.249.148:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:57:05.079648 28195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.068350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:33967 (local address 127.8.249.190:36295)
0926 20:57:05.068526 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:05.068531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:05.070122 (+ 1591us) server_negotiation.cc:407] Connection header received
0926 20:57:05.071992 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:05.071998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:05.072096 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:05.072219 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:05.073280 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.074071 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.076358 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.077057 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.077189 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.078775 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:05.078781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:05.078784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:05.078841 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:05.078885 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:05.078898 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:05.079015 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:05.079239 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 20:57:05.079340 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:57:05.080709 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.069023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36295 (local address 127.8.249.130:33967)
0926 20:57:05.069949 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:05.070000 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:05.071718 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:05.072448 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:05.072457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:05.072554 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:05.073111 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.073126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.074236 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.074247 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.076202 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.076216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.077300 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.077309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.077619 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.078432 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:57:05.078434 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:05.078485 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:57:05.078570 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:05.079015 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:05.079028 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:05.079038 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:57:05.079231 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 20:57:05.079669 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":297,"threads_started":1}
I20260926 20:57:05.082219 28324 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36295
I20260926 20:57:05.082554 28324 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:05.083381 28324 heartbeater.cc:511] Master 127.8.249.190:36295 requested a full tablet report, sending...
I20260926 20:57:05.084784 28007 ts_manager.cc:194] Registered new tserver with Master: 9075b946e81c497f85026e7134e66eb5 (127.8.249.130:32821)
I20260926 20:57:05.085614 28007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.249.130:33967
W20260926 20:57:05.412011 28327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:05.412293 28327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:05.412449 28327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:05.419809 28327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:05.419906 28327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:05.420109 28327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:57:05.429356 28327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:05.430649 28327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:05.432468 28327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:05.446630 28332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:05.446687 28333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:05.446714 28335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:05.447877 28327 server_base.cc:1034] running on GCE node
I20260926 20:57:05.448508 28327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:05.449453 28327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:05.450644 28327 hybrid_clock.cc:648] HybridClock initialized: now 1790456225450575 us; error 59 us; skew 500 ppm
I20260926 20:57:05.451031 28327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:57:05.453411 28327 webserver.cc:460] Webserver started at http://127.8.249.131:33589/ using document root <none> and password file <none>
I20260926 20:57:05.454165 28327 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:05.454281 28327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:05.454630 28327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:05.457693 28327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "e1781bb9ee6141028175f96698773781"
format_stamp: "Formatted at 2026-09-26 20:57:05 on dist-test-slave-2dg3"
server_key: "0160a14f19045a7a1444ceddd206abba"
server_key_iv: "8ffa164113ba9fe1d9edefdae5409051"
server_key_version: "encryptionkey@0"
I20260926 20:57:05.458552 28327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "e1781bb9ee6141028175f96698773781"
format_stamp: "Formatted at 2026-09-26 20:57:05 on dist-test-slave-2dg3"
server_key: "0160a14f19045a7a1444ceddd206abba"
server_key_iv: "8ffa164113ba9fe1d9edefdae5409051"
server_key_version: "encryptionkey@0"
I20260926 20:57:05.463938 28327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 20:57:05.467618 28341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:05.468988 28327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:57:05.469163 28327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "e1781bb9ee6141028175f96698773781"
format_stamp: "Formatted at 2026-09-26 20:57:05 on dist-test-slave-2dg3"
server_key: "0160a14f19045a7a1444ceddd206abba"
server_key_iv: "8ffa164113ba9fe1d9edefdae5409051"
server_key_version: "encryptionkey@0"
I20260926 20:57:05.469393 28327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:05.502404 28327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:05.503414 28327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:05.503723 28327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:05.504750 28327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:05.506255 28327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:05.506345 28327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:05.506553 28327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:05.506641 28327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:05.540331 28327 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:40719
I20260926 20:57:05.540426 28453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:40719 every 8 connection(s)
I20260926 20:57:05.541712 28327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:57:05.547487 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 28327
I20260926 20:57:05.547672 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:57:05.548200 9190 external_mini_cluster.cc:1428] Setting key 2b4a8b65332e70503e6ee4f7f82c8190
I20260926 20:57:05.555644 28195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.544492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:52593 (local address 127.8.249.190:36295)
0926 20:57:05.544706 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:05.544712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:05.546158 (+ 1446us) server_negotiation.cc:407] Connection header received
0926 20:57:05.548556 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:05.548561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:05.548639 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:05.548795 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:05.550291 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.551342 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.553045 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.553461 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.553568 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.554942 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:05.554947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:05.554950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:05.554968 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:05.555007 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:05.555019 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:05.555091 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:05.555331 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 20:57:05.555412 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 20:57:05.556643 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.545079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36295 (local address 127.8.249.131:52593)
0926 20:57:05.545985 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:05.546031 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:05.548201 (+ 2170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:05.549065 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:05.549077 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:05.549209 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:05.550084 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.550112 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.551497 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.551501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.552893 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.552909 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.553657 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.553661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.553856 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.554589 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:57:05.554591 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:05.554638 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:57:05.554716 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:05.555113 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:05.555121 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:05.555127 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:57:05.555270 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 20:57:05.555582 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":290,"threads_started":1}
I20260926 20:57:05.557940 28454 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36295
I20260926 20:57:05.558236 28454 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:05.559093 28454 heartbeater.cc:511] Master 127.8.249.190:36295 requested a full tablet report, sending...
I20260926 20:57:05.560477 28007 ts_manager.cc:194] Registered new tserver with Master: e1781bb9ee6141028175f96698773781 (127.8.249.131:40719)
I20260926 20:57:05.561226 28007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.249.131:52593
I20260926 20:57:05.563721 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:57:05.575590 28195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.569448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.8.249.190:36295)
0926 20:57:05.569623 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:05.569627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:05.570121 (+ 494us) server_negotiation.cc:407] Connection header received
0926 20:57:05.570252 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:05.570257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:05.570325 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:05.570408 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:05.570883 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.571581 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.573003 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.573472 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.573638 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.574782 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:05.574834 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:05.574840 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:05.574861 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:05.574901 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:05.574916 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:05.575064 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:05.575244 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 20:57:05.575298 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:57:05.579599 28007 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:57:05.586277 28007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:57:05.612493 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.601978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:40719 (local address 127.0.0.1:56476)
0926 20:57:05.602963 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:05.602979 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:05.603103 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:05.603699 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:05.603709 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:05.603735 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:05.604164 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.604176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.606340 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.606345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.608839 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.608853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.611407 (+ 2554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.611416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.611615 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.611670 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 20:57:05.611804 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 20:57:05.612053 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":28,"thread_start_us":684,"threads_started":1}
I20260926 20:57:05.612980 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.598341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:32821 (local address 127.0.0.1:36670)
0926 20:57:05.601591 (+ 3250us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:05.601631 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:05.601944 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:05.603132 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:05.603140 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:05.603183 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:05.603686 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.603700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.605676 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.605681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.610009 (+ 4328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.610021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.612159 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.612167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.612383 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.612415 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 20:57:05.612505 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:57:05.612644 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2122,"mutex_wait_us":1756,"thread_start_us":3146,"threads_started":1}
I20260926 20:57:05.613803 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.598516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:41235 (local address 127.0.0.1:58656)
0926 20:57:05.602465 (+ 3949us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:05.602484 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:05.602619 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:05.603366 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:05.603370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:05.603402 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:05.603969 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.603982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.607565 (+ 3583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.607570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.609714 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.609729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.612391 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.612400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.613233 (+ 833us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.613268 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:57:05.613350 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:57:05.613489 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3813,"mutex_wait_us":27,"thread_start_us":1009,"threads_started":1}
I20260926 20:57:05.613886 28468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.598591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36670 (local address 127.8.249.130:32821)
0926 20:57:05.600011 (+ 1420us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:05.600025 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:05.601765 (+ 1740us) server_negotiation.cc:407] Connection header received
0926 20:57:05.602210 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:05.602221 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:05.602566 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:05.602814 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:05.603942 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.605504 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.610205 (+ 4701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.611685 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.611807 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.612026 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:05.613355 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0926 20:57:05.613537 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":1489,"threads_started":1}
I20260926 20:57:05.614604 28470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.600083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.8.249.131:40719)
0926 20:57:05.601702 (+ 1619us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:05.601710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:05.603100 (+ 1390us) server_negotiation.cc:407] Connection header received
0926 20:57:05.603229 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:05.603236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:05.603487 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:05.603679 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:05.604345 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.606147 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.609395 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.610833 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.610982 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.611275 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:05.613800 (+ 2525us) server_negotiation.cc:299] Negotiation successful
0926 20:57:05.614246 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1450,"thread_start_us":1289,"threads_started":1}
I20260926 20:57:05.614598 28469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.598973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.8.249.129:41235)
0926 20:57:05.600856 (+ 1883us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:05.600874 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:05.602777 (+ 1903us) server_negotiation.cc:407] Connection header received
0926 20:57:05.602924 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:05.602931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:05.603182 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:05.603379 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:05.604166 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.607410 (+ 3244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.609930 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.611804 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.611992 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.612252 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:05.614057 (+ 1805us) server_negotiation.cc:299] Negotiation successful
0926 20:57:05.614219 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1738,"thread_start_us":2087,"threads_started":1}
I20260926 20:57:05.617684 28259 tablet_service.cc:1467] Processing CreateTablet for tablet 4d26a241fcdb45ca8be9d48f510d15ab (DEFAULT_TABLE table=test-table [id=0f3f523cd1a248908b99b365bb3a7ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:05.618237 28389 tablet_service.cc:1467] Processing CreateTablet for tablet 4d26a241fcdb45ca8be9d48f510d15ab (DEFAULT_TABLE table=test-table [id=0f3f523cd1a248908b99b365bb3a7ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:05.619302 28259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d26a241fcdb45ca8be9d48f510d15ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:05.619804 28389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d26a241fcdb45ca8be9d48f510d15ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:05.620446 28128 tablet_service.cc:1467] Processing CreateTablet for tablet 4d26a241fcdb45ca8be9d48f510d15ab (DEFAULT_TABLE table=test-table [id=0f3f523cd1a248908b99b365bb3a7ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:05.621482 28128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d26a241fcdb45ca8be9d48f510d15ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:05.631486 28193 heartbeater.cc:503] Master 127.8.249.190:36295 was elected leader, sending a full tablet report...
I20260926 20:57:05.634243 28472 tablet_bootstrap.cc:492] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb: Bootstrap starting.
I20260926 20:57:05.635625 28473 tablet_bootstrap.cc:492] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5: Bootstrap starting.
I20260926 20:57:05.637329 28474 tablet_bootstrap.cc:492] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781: Bootstrap starting.
I20260926 20:57:05.638603 28472 tablet_bootstrap.cc:654] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:05.639322 28473 tablet_bootstrap.cc:654] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:05.639971 28472 log.cc:824] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:05.640569 28473 log.cc:824] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:05.640617 28474 tablet_bootstrap.cc:654] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:05.641826 28474 log.cc:824] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:05.643875 28472 tablet_bootstrap.cc:492] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb: No bootstrap required, opened a new log
I20260926 20:57:05.644173 28472 ts_tablet_manager.cc:1397] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260926 20:57:05.644472 28473 tablet_bootstrap.cc:492] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5: No bootstrap required, opened a new log
I20260926 20:57:05.644747 28473 ts_tablet_manager.cc:1397] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 20:57:05.644999 28474 tablet_bootstrap.cc:492] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781: No bootstrap required, opened a new log
I20260926 20:57:05.645262 28474 ts_tablet_manager.cc:1397] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 20:57:05.651597 28472 raft_consensus.cc:348] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9075b946e81c497f85026e7134e66eb5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 32821 } } peers { permanent_uuid: "9523a2b5d6014f54bf8e89e420fd3aeb" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 41235 } } peers { permanent_uuid: "e1781bb9ee6141028175f96698773781" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40719 } }
I20260926 20:57:05.652068 28472 raft_consensus.cc:374] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:05.652192 28472 raft_consensus.cc:729] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9523a2b5d6014f54bf8e89e420fd3aeb, State: Initialized, Role: FOLLOWER
I20260926 20:57:05.652935 28472 consensus_queue.cc:260] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9075b946e81c497f85026e7134e66eb5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 32821 } } peers { permanent_uuid: "9523a2b5d6014f54bf8e89e420fd3aeb" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 41235 } } peers { permanent_uuid: "e1781bb9ee6141028175f96698773781" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40719 } }
I20260926 20:57:05.655495 28473 raft_consensus.cc:348] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9075b946e81c497f85026e7134e66eb5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 32821 } } peers { permanent_uuid: "9523a2b5d6014f54bf8e89e420fd3aeb" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 41235 } } peers { permanent_uuid: "e1781bb9ee6141028175f96698773781" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40719 } }
I20260926 20:57:05.655704 28474 raft_consensus.cc:348] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9075b946e81c497f85026e7134e66eb5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 32821 } } peers { permanent_uuid: "9523a2b5d6014f54bf8e89e420fd3aeb" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 41235 } } peers { permanent_uuid: "e1781bb9ee6141028175f96698773781" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40719 } }
I20260926 20:57:05.655977 28473 raft_consensus.cc:374] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:05.656071 28473 raft_consensus.cc:729] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9075b946e81c497f85026e7134e66eb5, State: Initialized, Role: FOLLOWER
I20260926 20:57:05.656184 28474 raft_consensus.cc:374] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:05.656312 28474 raft_consensus.cc:729] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1781bb9ee6141028175f96698773781, State: Initialized, Role: FOLLOWER
I20260926 20:57:05.656986 28472 ts_tablet_manager.cc:1428] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260926 20:57:05.656947 28473 consensus_queue.cc:260] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9075b946e81c497f85026e7134e66eb5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 32821 } } peers { permanent_uuid: "9523a2b5d6014f54bf8e89e420fd3aeb" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 41235 } } peers { permanent_uuid: "e1781bb9ee6141028175f96698773781" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40719 } }
I20260926 20:57:05.656948 28474 consensus_queue.cc:260] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9075b946e81c497f85026e7134e66eb5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 32821 } } peers { permanent_uuid: "9523a2b5d6014f54bf8e89e420fd3aeb" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 41235 } } peers { permanent_uuid: "e1781bb9ee6141028175f96698773781" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40719 } }
I20260926 20:57:05.658144 28454 heartbeater.cc:503] Master 127.8.249.190:36295 was elected leader, sending a full tablet report...
I20260926 20:57:05.659765 28474 ts_tablet_manager.cc:1428] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 20:57:05.660737 28324 heartbeater.cc:503] Master 127.8.249.190:36295 was elected leader, sending a full tablet report...
I20260926 20:57:05.661410 28473 ts_tablet_manager.cc:1428] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
I20260926 20:57:05.681654 28479 raft_consensus.cc:482] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:57:05.682019 28479 raft_consensus.cc:504] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9075b946e81c497f85026e7134e66eb5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 32821 } } peers { permanent_uuid: "9523a2b5d6014f54bf8e89e420fd3aeb" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 41235 } } peers { permanent_uuid: "e1781bb9ee6141028175f96698773781" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40719 } }
I20260926 20:57:05.683928 28479 leader_election.cc:288] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9075b946e81c497f85026e7134e66eb5 (127.8.249.130:32821), 9523a2b5d6014f54bf8e89e420fd3aeb (127.8.249.129:41235)
I20260926 20:57:05.690657 28468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.684384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:47339 (local address 127.8.249.130:32821)
0926 20:57:05.684579 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:05.684587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:05.684686 (+ 99us) server_negotiation.cc:407] Connection header received
0926 20:57:05.684980 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:05.684988 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:05.685083 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:05.685183 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:05.685774 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.686720 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.688764 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.689679 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.689799 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.689849 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:05.690222 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 20:57:05.690369 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:57:05.690716 28456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.684219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:32821 (local address 127.8.249.131:47339)
0926 20:57:05.684563 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:05.684592 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:05.684756 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:05.685271 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:05.685296 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:05.685330 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:05.685579 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.685594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.686888 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.686894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.688593 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.688611 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.689849 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.689856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.690044 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.690089 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:57:05.690204 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:57:05.690380 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":26,"spinlock_wait_cycles":32128}
I20260926 20:57:05.691181 28469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.684480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:57275 (local address 127.8.249.129:41235)
0926 20:57:05.684686 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:05.684692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:05.685328 (+ 636us) server_negotiation.cc:407] Connection header received
0926 20:57:05.685559 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:05.685565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:05.685667 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:05.685782 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:05.686386 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.687338 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.689200 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.690193 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.690361 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.690434 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:05.690724 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 20:57:05.690899 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:57:05.691712 28481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.684384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:41235 (local address 127.8.249.131:57275)
0926 20:57:05.685213 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:05.685229 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:05.685416 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:05.685795 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:05.685799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:05.685824 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:05.686214 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.686228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.687497 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.687501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.689007 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:05.689023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.690360 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.690366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:05.690546 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.690576 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 20:57:05.690672 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 20:57:05.691451 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":2,"spinlock_wait_cycles":39808,"thread_start_us":479,"threads_started":1}
I20260926 20:57:05.691754 28279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d26a241fcdb45ca8be9d48f510d15ab" candidate_uuid: "e1781bb9ee6141028175f96698773781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9075b946e81c497f85026e7134e66eb5" is_pre_election: true
I20260926 20:57:05.692377 28279 raft_consensus.cc:2454] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1781bb9ee6141028175f96698773781 in term 0.
I20260926 20:57:05.692876 28148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d26a241fcdb45ca8be9d48f510d15ab" candidate_uuid: "e1781bb9ee6141028175f96698773781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9523a2b5d6014f54bf8e89e420fd3aeb" is_pre_election: true
I20260926 20:57:05.693183 28343 leader_election.cc:302] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9075b946e81c497f85026e7134e66eb5, e1781bb9ee6141028175f96698773781; no voters:
I20260926 20:57:05.693385 28148 raft_consensus.cc:2454] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1781bb9ee6141028175f96698773781 in term 0.
I20260926 20:57:05.693625 28479 raft_consensus.cc:2781] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:57:05.693782 28479 raft_consensus.cc:482] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:57:05.693912 28479 raft_consensus.cc:3037] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:05.696342 28479 raft_consensus.cc:504] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9075b946e81c497f85026e7134e66eb5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 32821 } } peers { permanent_uuid: "9523a2b5d6014f54bf8e89e420fd3aeb" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 41235 } } peers { permanent_uuid: "e1781bb9ee6141028175f96698773781" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40719 } }
I20260926 20:57:05.696987 28479 leader_election.cc:288] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [CANDIDATE]: Term 1 election: Requested vote from peers 9075b946e81c497f85026e7134e66eb5 (127.8.249.130:32821), 9523a2b5d6014f54bf8e89e420fd3aeb (127.8.249.129:41235)
I20260926 20:57:05.697412 28279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d26a241fcdb45ca8be9d48f510d15ab" candidate_uuid: "e1781bb9ee6141028175f96698773781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9075b946e81c497f85026e7134e66eb5"
I20260926 20:57:05.697489 28148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d26a241fcdb45ca8be9d48f510d15ab" candidate_uuid: "e1781bb9ee6141028175f96698773781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9523a2b5d6014f54bf8e89e420fd3aeb"
I20260926 20:57:05.697638 28279 raft_consensus.cc:3037] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:05.697721 28148 raft_consensus.cc:3037] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:05.699987 28148 raft_consensus.cc:2454] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1781bb9ee6141028175f96698773781 in term 1.
I20260926 20:57:05.699994 28279 raft_consensus.cc:2454] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1781bb9ee6141028175f96698773781 in term 1.
I20260926 20:57:05.700407 28345 leader_election.cc:302] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9523a2b5d6014f54bf8e89e420fd3aeb, e1781bb9ee6141028175f96698773781; no voters:
I20260926 20:57:05.700721 28479 raft_consensus.cc:2781] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:05.701148 28479 raft_consensus.cc:686] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [term 1 LEADER]: Becoming Leader. State: Replica: e1781bb9ee6141028175f96698773781, State: Running, Role: LEADER
I20260926 20:57:05.701638 28479 consensus_queue.cc:237] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9075b946e81c497f85026e7134e66eb5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 32821 } } peers { permanent_uuid: "9523a2b5d6014f54bf8e89e420fd3aeb" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 41235 } } peers { permanent_uuid: "e1781bb9ee6141028175f96698773781" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40719 } }
I20260926 20:57:05.706717 28007 catalog_manager.cc:5495] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1781bb9ee6141028175f96698773781 (127.8.249.131). New cstate: current_term: 1 leader_uuid: "e1781bb9ee6141028175f96698773781" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9075b946e81c497f85026e7134e66eb5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 32821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9523a2b5d6014f54bf8e89e420fd3aeb" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1781bb9ee6141028175f96698773781" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40719 } health_report { overall_health: HEALTHY } } }
I20260926 20:57:05.739151 28470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.734258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.8.249.131:40719)
0926 20:57:05.734424 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:05.734428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:05.734448 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:57:05.734635 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:05.734639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:05.734728 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:05.734883 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:05.735204 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.735910 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.737204 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.737612 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.737698 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.738139 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:05.738623 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:05.738749 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:05.738922 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 20:57:05.738959 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:57:05.747221 28279 raft_consensus.cc:1261] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5 [term 1 FOLLOWER]: Refusing update from remote peer e1781bb9ee6141028175f96698773781: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:57:05.747222 28148 raft_consensus.cc:1261] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb [term 1 FOLLOWER]: Refusing update from remote peer e1781bb9ee6141028175f96698773781: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:57:05.748202 28479 consensus_queue.cc:1035] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9523a2b5d6014f54bf8e89e420fd3aeb" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:57:05.748479 28482 consensus_queue.cc:1035] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9075b946e81c497f85026e7134e66eb5" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 32821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:57:05.772629 28007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:57:05.772909 28007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:57:05.776731 28007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b5af162d09e4f3ca882bca09ed64257: Sending DeleteTablet for 3 replicas of tablet 4d26a241fcdb45ca8be9d48f510d15ab
I20260926 20:57:05.778172 28259 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d26a241fcdb45ca8be9d48f510d15ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:05 UTC) from {username='slave'} at 127.0.0.1:36670
I20260926 20:57:05.778285 28128 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d26a241fcdb45ca8be9d48f510d15ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:05 UTC) from {username='slave'} at 127.0.0.1:58656
I20260926 20:57:05.779290 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 28066
I20260926 20:57:05.779275 28389 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d26a241fcdb45ca8be9d48f510d15ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:05 UTC) from {username='slave'} at 127.0.0.1:56476
I20260926 20:57:05.790835 28496 tablet_replica.cc:330] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5: stopping tablet replica
I20260926 20:57:05.792004 28495 tablet_replica.cc:330] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb: stopping tablet replica
I20260926 20:57:05.792553 28495 raft_consensus.cc:2229] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:57:05.793136 28495 raft_consensus.cc:2258] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:57:05.793514 28496 raft_consensus.cc:2229] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:57:05.794291 28496 raft_consensus.cc:2258] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:57:05.797204 28496 ts_tablet_manager.cc:1905] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:05.797390 28497 tablet_replica.cc:330] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781: stopping tablet replica
I20260926 20:57:05.797921 28497 raft_consensus.cc:2229] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:57:05.798671 28497 raft_consensus.cc:2258] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:57:05.800871 28497 ts_tablet_manager.cc:1905] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:05.801210 28469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.780328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.8.249.129:41235)
0926 20:57:05.780507 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:05.780514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:05.791632 (+ 11118us) server_negotiation.cc:407] Connection header received
0926 20:57:05.791976 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:05.791989 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:05.792160 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:05.792258 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:05.793916 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.795113 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.797128 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.797798 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.797909 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.799490 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:05.799502 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:05.799521 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:05.800268 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:05.800345 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:05.800370 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:05.800470 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:05.800720 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 20:57:05.800839 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:57:05.802386 28188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:05.804723 28496 ts_tablet_manager.cc:1918] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:57:05.804970 28496 log.cc:1196] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/4d26a241fcdb45ca8be9d48f510d15ab
I20260926 20:57:05.805665 28496 ts_tablet_manager.cc:1939] T 4d26a241fcdb45ca8be9d48f510d15ab P 9075b946e81c497f85026e7134e66eb5: Deleting consensus metadata
I20260926 20:57:05.808259 27992 catalog_manager.cc:4841] TS 9075b946e81c497f85026e7134e66eb5 (127.8.249.130:32821): tablet 4d26a241fcdb45ca8be9d48f510d15ab (table test-table [id=0f3f523cd1a248908b99b365bb3a7ac1]) successfully deleted
I20260926 20:57:05.809624 28497 ts_tablet_manager.cc:1918] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:57:05.809857 28497 log.cc:1196] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/4d26a241fcdb45ca8be9d48f510d15ab
I20260926 20:57:05.810428 28497 ts_tablet_manager.cc:1939] T 4d26a241fcdb45ca8be9d48f510d15ab P e1781bb9ee6141028175f96698773781: Deleting consensus metadata
I20260926 20:57:05.812409 27993 catalog_manager.cc:4841] TS e1781bb9ee6141028175f96698773781 (127.8.249.131:40719): tablet 4d26a241fcdb45ca8be9d48f510d15ab (table test-table [id=0f3f523cd1a248908b99b365bb3a7ac1]) successfully deleted
I20260926 20:57:05.803910 28495 ts_tablet_manager.cc:1905] T 4d26a241fcdb45ca8be9d48f510d15ab P 9523a2b5d6014f54bf8e89e420fd3aeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:05.895264 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 28066
W20260926 20:57:05.920755 27994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.249.129:41235 (error 108)
I20260926 20:57:05.921518 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 28197
I20260926 20:57:05.922961 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.922338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:41235 (local address 127.0.0.1:58672)
0926 20:57:05.922575 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:05.922739 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:41235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260926 20:57:05.923273 27994 catalog_manager.cc:4584] TS 9523a2b5d6014f54bf8e89e420fd3aeb (127.8.249.129:41235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d26a241fcdb45ca8be9d48f510d15ab: Network error: Client connection negotiation failed: client connection to 127.8.249.129:41235: connect: Connection refused (error 111)
I20260926 20:57:05.929751 28468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.922394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.8.249.130:32821)
0926 20:57:05.922565 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:05.922571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:05.922653 (+ 82us) server_negotiation.cc:407] Connection header received
0926 20:57:05.922875 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:05.922880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:05.923018 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:05.923164 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:05.923788 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.924948 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.926232 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:05.926920 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:05.927051 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:05.928131 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:05.928143 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:05.928162 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:05.928957 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:05.929021 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:05.929046 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:05.929167 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:05.929283 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 20:57:05.929398 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:57:05.930509 28319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:05.977198 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.976626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:41235 (local address 127.0.0.1:58684)
0926 20:57:05.976876 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:05.977000 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:41235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 20:57:05.978397 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:05.977971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:41235 (local address 127.0.0.1:58690)
0926 20:57:05.978141 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:05.978240 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:41235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 20:57:06.020021 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 28197
I20260926 20:57:06.036825 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 28327
I20260926 20:57:06.039168 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:06.038597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:41235 (local address 127.0.0.1:58698)
0926 20:57:06.038873 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:06.038980 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:41235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 20:57:06.040346 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:06.039897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:41235 (local address 127.0.0.1:58704)
0926 20:57:06.040063 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:06.040155 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:41235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:57:06.057909 28470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:06.037705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.8.249.131:40719)
0926 20:57:06.037884 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:06.037892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:06.037916 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:57:06.038160 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:06.038166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:06.038252 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:06.038376 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:06.038853 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:06.040026 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:06.041615 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:06.042138 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:06.042289 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:06.056358 (+ 14069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:06.056369 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:06.056383 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:06.057017 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:06.057077 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:06.057102 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:06.057249 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:06.057428 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:57:06.057545 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 20:57:06.058670 28449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:06.122037 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:06.121547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:41235 (local address 127.0.0.1:58708)
0926 20:57:06.121770 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:06.121882 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:41235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 20:57:06.123172 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:06.122760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:41235 (local address 127.0.0.1:58710)
0926 20:57:06.122994 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:06.123067 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:41235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260926 20:57:06.181481 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 28327
I20260926 20:57:06.201670 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 27976
I20260926 20:57:06.202778 28037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:06.311602 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:06.310946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:41235 (local address 127.0.0.1:58720)
0926 20:57:06.311198 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:06.311342 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:41235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 20:57:06.311633 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 27976
2026-09-26T20:57:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2739 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:06 dist-test-slave-2dg3 krb5kdc[28506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:06 dist-test-slave-2dg3 krb5kdc[28506](info): set up 2 sockets
Sep 26 20:57:06 dist-test-slave-2dg3 krb5kdc[28506](info): commencing operation
krb5kdc: starting...
W20260926 20:57:07.431133 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:07 dist-test-slave-2dg3 krb5kdc[28506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:57:07.687325 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36715
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:36715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:08.020628 28519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:08.020975 28519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:08.021135 28519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:08.028545 28519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:08.028724 28519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:08.028838 28519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:08.028936 28519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:08.029026 28519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:08.038932 28519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:36715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:08.040274 28519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:08.041618 28519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:08.053547 28524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:08.053542 28527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:08.053551 28525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:08.054518 28519 server_base.cc:1034] running on GCE node
I20260926 20:57:08.055480 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:08.056910 28519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:08.058115 28519 hybrid_clock.cc:648] HybridClock initialized: now 1790456228058050 us; error 48 us; skew 500 ppm
I20260926 20:57:08.058496 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:08 dist-test-slave-2dg3 krb5kdc[28506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456228, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:08.061707 28519 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:57:08.063282 28519 webserver.cc:460] Webserver started at http://127.8.249.190:44637/ using document root <none> and password file <none>
I20260926 20:57:08.063993 28519 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:08.064142 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:08.064465 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:08.067536 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "d528ae8abd8f4dc2975ca4d09c6ff4a2"
format_stamp: "Formatted at 2026-09-26 20:57:08 on dist-test-slave-2dg3"
server_key: "6034e8fb5863aad08989755f6b9d308f"
server_key_iv: "731ef63fd085c308fb44a0dafc6b88e4"
server_key_version: "encryptionkey@0"
I20260926 20:57:08.068344 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "d528ae8abd8f4dc2975ca4d09c6ff4a2"
format_stamp: "Formatted at 2026-09-26 20:57:08 on dist-test-slave-2dg3"
server_key: "6034e8fb5863aad08989755f6b9d308f"
server_key_iv: "731ef63fd085c308fb44a0dafc6b88e4"
server_key_version: "encryptionkey@0"
I20260926 20:57:08.073909 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:57:08.077565 28534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:08.079018 28519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 20:57:08.079228 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "d528ae8abd8f4dc2975ca4d09c6ff4a2"
format_stamp: "Formatted at 2026-09-26 20:57:08 on dist-test-slave-2dg3"
server_key: "6034e8fb5863aad08989755f6b9d308f"
server_key_iv: "731ef63fd085c308fb44a0dafc6b88e4"
server_key_version: "encryptionkey@0"
I20260926 20:57:08.079447 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:08.097695 28519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:08.102026 28519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:08.102339 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:08.121467 28519 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:36715
I20260926 20:57:08.121484 28585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:36715 every 8 connection(s)
I20260926 20:57:08.122998 28519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:08.126659 28586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:08.132365 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 28519
I20260926 20:57:08.132560 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:08.133004 9190 external_mini_cluster.cc:1428] Setting key 4a1ec2d1724980faa3a35f7541b71aa5
I20260926 20:57:08.137303 28586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2: Bootstrap starting.
I20260926 20:57:08.141069 28586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:08.142233 28586 log.cc:824] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:08.145432 28586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2: No bootstrap required, opened a new log
Sep 26 20:57:08 dist-test-slave-2dg3 krb5kdc[28506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:08.156117 28586 raft_consensus.cc:348] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d528ae8abd8f4dc2975ca4d09c6ff4a2" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36715 } }
I20260926 20:57:08.156672 28586 raft_consensus.cc:374] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:08.156841 28586 raft_consensus.cc:729] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d528ae8abd8f4dc2975ca4d09c6ff4a2, State: Initialized, Role: FOLLOWER
I20260926 20:57:08.157538 28586 consensus_queue.cc:260] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d528ae8abd8f4dc2975ca4d09c6ff4a2" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36715 } }
I20260926 20:57:08.157840 28586 raft_consensus.cc:388] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:08.157984 28586 raft_consensus.cc:482] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:08.158164 28586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:08.158480 28589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:08.134930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.8.249.190:36715)
0926 20:57:08.135735 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:08.135746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:08.135792 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:57:08.137594 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:08.137616 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:08.138012 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:08.138328 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:08.140286 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:08.141634 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:08.143400 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:08.144084 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:08.144222 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:08.149137 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:08.149154 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:08.149179 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:08.149242 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:08.153550 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:08.154340 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:08.154351 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:08.154360 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:08.154595 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:08.155044 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:08.155048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:08.155052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:08.155812 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:08.156164 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:08.156501 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 20:57:08.157019 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":395,"threads_started":1}
I20260926 20:57:08.160694 28586 raft_consensus.cc:504] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d528ae8abd8f4dc2975ca4d09c6ff4a2" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36715 } }
I20260926 20:57:08.161204 28586 leader_election.cc:302] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d528ae8abd8f4dc2975ca4d09c6ff4a2; no voters:
I20260926 20:57:08.161895 28586 leader_election.cc:288] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:08.162014 28591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:08.162634 28591 raft_consensus.cc:686] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [term 1 LEADER]: Becoming Leader. State: Replica: d528ae8abd8f4dc2975ca4d09c6ff4a2, State: Running, Role: LEADER
I20260926 20:57:08.163273 28591 consensus_queue.cc:237] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d528ae8abd8f4dc2975ca4d09c6ff4a2" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36715 } }
I20260926 20:57:08.164180 28586 sys_catalog.cc:564] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:08.166170 28592 sys_catalog.cc:455] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d528ae8abd8f4dc2975ca4d09c6ff4a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d528ae8abd8f4dc2975ca4d09c6ff4a2" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36715 } } }
I20260926 20:57:08.166767 28592 sys_catalog.cc:458] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:08.166251 28593 sys_catalog.cc:455] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d528ae8abd8f4dc2975ca4d09c6ff4a2. Latest consensus state: current_term: 1 leader_uuid: "d528ae8abd8f4dc2975ca4d09c6ff4a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d528ae8abd8f4dc2975ca4d09c6ff4a2" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36715 } } }
I20260926 20:57:08.167618 28593 sys_catalog.cc:458] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:08.173990 28596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:08.182047 28596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:08.192391 28596 catalog_manager.cc:1347] Generated new cluster ID: 82a53ea38bbe46c79dc78311419867c2
I20260926 20:57:08.192536 28596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:08.210258 28596 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:08.212280 28596 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:08.225618 28596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2: Generated new TSK 0
I20260926 20:57:08.226464 28596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
Entry for principal kudu/127.8.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab.
I20260926 20:57:08.292416 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36715
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:08.630119 28612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:08.630499 28612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:08.630674 28612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:08.638247 28612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:08.638417 28612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:08.638619 28612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:57:08.649013 28612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.129.keytab
--principal=kudu/127.8.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:08.650417 28612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:08.651901 28612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:08.665927 28617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:08.665977 28618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:08.666184 28620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:08.667062 28612 server_base.cc:1034] running on GCE node
I20260926 20:57:08.667802 28612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:08.669063 28612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:08.670311 28612 hybrid_clock.cc:648] HybridClock initialized: now 1790456228670190 us; error 109 us; skew 500 ppm
I20260926 20:57:08.670686 28612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:08 dist-test-slave-2dg3 krb5kdc[28506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456228, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:08.674086 28612 init.cc:377] Logged in from keytab as kudu/127.8.249.129@KRBTEST.COM (short username kudu)
I20260926 20:57:08.675777 28612 webserver.cc:460] Webserver started at http://127.8.249.129:33719/ using document root <none> and password file <none>
I20260926 20:57:08.676532 28612 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:08.676679 28612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:08.676999 28612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:08.679983 28612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "bd202ce78a8f4336b8a0a81481f6ed1d"
format_stamp: "Formatted at 2026-09-26 20:57:08 on dist-test-slave-2dg3"
server_key: "2137f098f9676916812e0d9e9b2ad988"
server_key_iv: "629e073d6501b8ea8f2ee12c50d32d99"
server_key_version: "encryptionkey@0"
I20260926 20:57:08.680792 28612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "bd202ce78a8f4336b8a0a81481f6ed1d"
format_stamp: "Formatted at 2026-09-26 20:57:08 on dist-test-slave-2dg3"
server_key: "2137f098f9676916812e0d9e9b2ad988"
server_key_iv: "629e073d6501b8ea8f2ee12c50d32d99"
server_key_version: "encryptionkey@0"
I20260926 20:57:08.686230 28612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 20:57:08.689718 28627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:08.691093 28612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:08.691258 28612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "bd202ce78a8f4336b8a0a81481f6ed1d"
format_stamp: "Formatted at 2026-09-26 20:57:08 on dist-test-slave-2dg3"
server_key: "2137f098f9676916812e0d9e9b2ad988"
server_key_iv: "629e073d6501b8ea8f2ee12c50d32d99"
server_key_version: "encryptionkey@0"
I20260926 20:57:08.691457 28612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:08.728390 28612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:08.732908 28612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:08.733217 28612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:08.734246 28612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:08.735745 28612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:08.735859 28612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:08.736008 28612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:08.736085 28612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:08.769389 28612 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:36499
I20260926 20:57:08.769435 28739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:36499 every 8 connection(s)
I20260926 20:57:08.770658 28612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:57:08.771605 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 28612
I20260926 20:57:08.771786 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:57:08.772373 9190 external_mini_cluster.cc:1428] Setting key 0b1ddab2d34d433cab0427b4b100f3a2
Sep 26 20:57:08 dist-test-slave-2dg3 krb5kdc[28506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456228, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.129@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:08.795379 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:08.773647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:40757 (local address 127.8.249.190:36715)
0926 20:57:08.774193 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:08.774211 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:08.775388 (+ 1177us) server_negotiation.cc:407] Connection header received
0926 20:57:08.778502 (+ 3114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:08.778506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:08.778571 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:08.778661 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:08.780994 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:08.781746 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:08.784406 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:08.784798 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:08.784932 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:08.788743 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:08.788749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:08.788752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:08.788790 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:08.791641 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:08.792557 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:08.792563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:08.792566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:08.792717 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:08.793284 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:08.793294 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:08.793297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:08.793594 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:08.793798 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:08.794644 (+ 846us) server_negotiation.cc:299] Negotiation successful
0926 20:57:08.794955 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":282,"threads_started":1}
I20260926 20:57:08.796490 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:08.774295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36715 (local address 127.8.249.129:40757)
0926 20:57:08.775191 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:08.775244 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:08.778188 (+ 2944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:08.778993 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:08.779009 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:08.779830 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:08.780673 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:08.780721 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:08.781924 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:08.781932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:08.784254 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:08.784274 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:08.785060 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:08.785067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:08.785288 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:08.786057 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:08.786083 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:08.788462 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:08.791843 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:08.791854 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:08.791875 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:08.792394 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:08.792849 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:08.792855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:08.792859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:08.793087 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:08.793768 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:08.793777 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:08.794169 (+ 392us) client_negotiation.cc:769] Sending connection context
0926 20:57:08.794519 (+ 350us) client_negotiation.cc:240] Negotiation successful
0926 20:57:08.795138 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":341,"threads_started":1}
I20260926 20:57:08.798432 28740 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36715
I20260926 20:57:08.798774 28740 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.249.130@KRBTEST.COM; defaulting to no policy
I20260926 20:57:08.799573 28740 heartbeater.cc:511] Master 127.8.249.190:36715 requested a full tablet report, sending...
I20260926 20:57:08.801975 28551 ts_manager.cc:194] Registered new tserver with Master: bd202ce78a8f4336b8a0a81481f6ed1d (127.8.249.129:36499)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.130@KRBTEST.COM" created.
I20260926 20:57:08.804282 28551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.129@KRBTEST.COM'} at 127.8.249.129:40757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
Entry for principal kudu/127.8.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab.
I20260926 20:57:08.833698 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36715
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:09.162360 28746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:09.162770 28746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:09.163008 28746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:09.170552 28746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:09.170722 28746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:09.170953 28746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:57:09.180917 28746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.130.keytab
--principal=kudu/127.8.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:09.182262 28746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:09.183768 28746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:09.196779 28751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:09.196722 28752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:09.196830 28754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:09.197888 28746 server_base.cc:1034] running on GCE node
I20260926 20:57:09.198500 28746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:09.199510 28746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:09.200714 28746 hybrid_clock.cc:648] HybridClock initialized: now 1790456229200662 us; error 40 us; skew 500 ppm
I20260926 20:57:09.201041 28746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:09 dist-test-slave-2dg3 krb5kdc[28506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456229, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:09.204237 28746 init.cc:377] Logged in from keytab as kudu/127.8.249.130@KRBTEST.COM (short username kudu)
I20260926 20:57:09.205619 28746 webserver.cc:460] Webserver started at http://127.8.249.130:44857/ using document root <none> and password file <none>
I20260926 20:57:09.206254 28746 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:09.206355 28746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:09.206681 28746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:09.210078 28746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "b9454db3d34c4a3cb6b67bd3912cc234"
format_stamp: "Formatted at 2026-09-26 20:57:09 on dist-test-slave-2dg3"
server_key: "70e46bb110c8af7f9c0489d397cc578d"
server_key_iv: "199b4de32d0a99825b7e32db9549b96c"
server_key_version: "encryptionkey@0"
I20260926 20:57:09.211305 28746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "b9454db3d34c4a3cb6b67bd3912cc234"
format_stamp: "Formatted at 2026-09-26 20:57:09 on dist-test-slave-2dg3"
server_key: "70e46bb110c8af7f9c0489d397cc578d"
server_key_iv: "199b4de32d0a99825b7e32db9549b96c"
server_key_version: "encryptionkey@0"
I20260926 20:57:09.217801 28746 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 20:57:09.221328 28761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:09.222656 28746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:57:09.222886 28746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "b9454db3d34c4a3cb6b67bd3912cc234"
format_stamp: "Formatted at 2026-09-26 20:57:09 on dist-test-slave-2dg3"
server_key: "70e46bb110c8af7f9c0489d397cc578d"
server_key_iv: "199b4de32d0a99825b7e32db9549b96c"
server_key_version: "encryptionkey@0"
I20260926 20:57:09.223114 28746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:09.240883 28746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:09.245455 28746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:09.245766 28746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:09.246764 28746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:09.248279 28746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:09.248384 28746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:09.248539 28746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:09.248611 28746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:09.281838 28746 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:39435
I20260926 20:57:09.281888 28873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:39435 every 8 connection(s)
I20260926 20:57:09.283319 28746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:57:09.288159 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 28746
I20260926 20:57:09.288339 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:57:09.288789 9190 external_mini_cluster.cc:1428] Setting key 5ace419b3ae28555b62ea3f9bde67da7
Sep 26 20:57:09 dist-test-slave-2dg3 krb5kdc[28506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456229, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.130@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:09.304369 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.285832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:53619 (local address 127.8.249.190:36715)
0926 20:57:09.286005 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:09.286009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:09.287216 (+ 1207us) server_negotiation.cc:407] Connection header received
0926 20:57:09.289303 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:09.289309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:09.289384 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:09.289470 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:09.291207 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.292301 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.293845 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.294600 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.294733 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.298283 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:09.298292 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:09.298300 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:09.298355 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:09.301124 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:09.302051 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:09.302055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:09.302057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:09.302173 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:09.302656 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:09.302659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:09.302660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:09.302981 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:09.303090 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:09.303954 (+ 864us) server_negotiation.cc:299] Negotiation successful
0926 20:57:09.304128 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:57:09.305941 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.286245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36715 (local address 127.8.249.130:53619)
0926 20:57:09.287056 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:09.287100 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:09.289014 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:09.289745 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:09.289754 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:09.290273 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:09.291035 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.291057 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.292484 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.292489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.293696 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.293708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.294901 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.294908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.295167 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.295884 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:09.295956 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:09.298055 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:09.301265 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:09.301276 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:09.301296 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:09.301933 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:09.302282 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:09.302289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:09.302296 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:09.302548 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:09.303104 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:09.303115 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:09.303567 (+ 452us) client_negotiation.cc:769] Sending connection context
0926 20:57:09.303872 (+ 305us) client_negotiation.cc:240] Negotiation successful
0926 20:57:09.304440 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":292,"threads_started":1}
I20260926 20:57:09.307493 28874 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36715
I20260926 20:57:09.307802 28874 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:09.308511 28874 heartbeater.cc:511] Master 127.8.249.190:36715 requested a full tablet report, sending...
I20260926 20:57:09.309989 28551 ts_manager.cc:194] Registered new tserver with Master: b9454db3d34c4a3cb6b67bd3912cc234 (127.8.249.130:39435)
I20260926 20:57:09.311317 28551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.130@KRBTEST.COM'} at 127.8.249.130:53619
WARNING: no policy specified for kudu/127.8.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
Entry for principal kudu/127.8.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab.
I20260926 20:57:09.347524 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36715
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:09.673430 28879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:09.673748 28879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:09.673903 28879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:09.681105 28879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:09.681217 28879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:09.681402 28879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:57:09.691283 28879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.131.keytab
--principal=kudu/127.8.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:36715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:09.692617 28879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:09.694016 28879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:09.707548 28887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:09.707558 28885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:09.707836 28884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:09.708398 28879 server_base.cc:1034] running on GCE node
I20260926 20:57:09.709131 28879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:09.710168 28879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:09.711365 28879 hybrid_clock.cc:648] HybridClock initialized: now 1790456229711301 us; error 72 us; skew 500 ppm
I20260926 20:57:09.711695 28879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:09 dist-test-slave-2dg3 krb5kdc[28506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456229, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:09.714979 28879 init.cc:377] Logged in from keytab as kudu/127.8.249.131@KRBTEST.COM (short username kudu)
I20260926 20:57:09.716492 28879 webserver.cc:460] Webserver started at http://127.8.249.131:34453/ using document root <none> and password file <none>
I20260926 20:57:09.717226 28879 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:09.717386 28879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:09.717715 28879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:09.720990 28879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "19227c15190945cd9ebc5dd8e8ee528b"
format_stamp: "Formatted at 2026-09-26 20:57:09 on dist-test-slave-2dg3"
server_key: "30c80687335117170d7b42c04916afd5"
server_key_iv: "0f7f9c5a8f73d3f5f6b03f0f5b5d94a3"
server_key_version: "encryptionkey@0"
I20260926 20:57:09.721810 28879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "19227c15190945cd9ebc5dd8e8ee528b"
format_stamp: "Formatted at 2026-09-26 20:57:09 on dist-test-slave-2dg3"
server_key: "30c80687335117170d7b42c04916afd5"
server_key_iv: "0f7f9c5a8f73d3f5f6b03f0f5b5d94a3"
server_key_version: "encryptionkey@0"
I20260926 20:57:09.727116 28879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:57:09.730577 28894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:09.732034 28879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:09.732249 28879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "19227c15190945cd9ebc5dd8e8ee528b"
format_stamp: "Formatted at 2026-09-26 20:57:09 on dist-test-slave-2dg3"
server_key: "30c80687335117170d7b42c04916afd5"
server_key_iv: "0f7f9c5a8f73d3f5f6b03f0f5b5d94a3"
server_key_version: "encryptionkey@0"
I20260926 20:57:09.732476 28879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:09.749819 28879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:09.754271 28879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:09.754581 28879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:09.755687 28879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:09.757057 28879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:09.757162 28879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:09.757323 28879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:09.757396 28879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:09.791585 28879 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:40071
I20260926 20:57:09.791607 29006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:40071 every 8 connection(s)
I20260926 20:57:09.792923 28879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:57:09.802297 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 28879
I20260926 20:57:09.802488 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:57:09.803056 9190 external_mini_cluster.cc:1428] Setting key 1ae22cad197b3d3d275168ea633c85ff
Sep 26 20:57:09 dist-test-slave-2dg3 krb5kdc[28506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456229, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.131@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:09.808223 28740 heartbeater.cc:503] Master 127.8.249.190:36715 was elected leader, sending a full tablet report...
I20260926 20:57:09.813374 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.795560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:56141 (local address 127.8.249.190:36715)
0926 20:57:09.795744 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:09.795749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:09.796902 (+ 1153us) server_negotiation.cc:407] Connection header received
0926 20:57:09.798755 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:09.798761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:09.798878 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:09.798994 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:09.800501 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.801239 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.803375 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.803811 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.803928 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.807888 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:09.807893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:09.807897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:09.807929 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:09.810746 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:09.811456 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:09.811460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:09.811462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:09.811583 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:09.811957 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:09.811960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:09.811962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:09.812232 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:09.812350 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:09.812961 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 20:57:09.813126 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:57:09.814388 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.796090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36715 (local address 127.8.249.131:56141)
0926 20:57:09.796748 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:09.796786 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:09.798459 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:09.799237 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:09.799246 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:09.799766 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:09.800317 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.800345 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.801411 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.801414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.803220 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.803236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.804047 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.804053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.804298 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.805116 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:09.805167 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:09.807703 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:09.810936 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:09.810943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:09.810955 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:09.811354 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:09.811674 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:09.811677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:09.811680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:09.811867 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:09.812333 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:09.812339 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:09.812645 (+ 306us) client_negotiation.cc:769] Sending connection context
0926 20:57:09.812878 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 20:57:09.813251 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":272,"threads_started":1}
I20260926 20:57:09.815929 29007 heartbeater.cc:347] Connected to a master server at 127.8.249.190:36715
I20260926 20:57:09.816344 29007 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:09.817205 29007 heartbeater.cc:511] Master 127.8.249.190:36715 requested a full tablet report, sending...
I20260926 20:57:09.818868 28551 ts_manager.cc:194] Registered new tserver with Master: 19227c15190945cd9ebc5dd8e8ee528b (127.8.249.131:40071)
I20260926 20:57:09.819676 28551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.249.131@KRBTEST.COM'} at 127.8.249.131:56141
I20260926 20:57:09.830979 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:57:09.847141 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.835182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.8.249.190:36715)
0926 20:57:09.835359 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:09.835364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:09.835853 (+ 489us) server_negotiation.cc:407] Connection header received
0926 20:57:09.836009 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:09.836013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:09.836078 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:09.836163 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:09.837570 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.838275 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.839593 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.839971 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.840063 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.842341 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:09.842346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:09.842349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:09.842381 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:09.844943 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:09.845523 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:09.845526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:09.845529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:09.845652 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:09.845938 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:09.845941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:09.845942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:09.846252 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:09.846437 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:09.846693 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 20:57:09.846920 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:57:09.850960 28551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:57:09.857869 28551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:57:09.883826 29020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.870833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:36499 (local address 127.0.0.1:40866)
0926 20:57:09.873480 (+ 2647us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:09.873499 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:09.873625 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:09.874729 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:09.874737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:09.874769 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:09.875250 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.875263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.877487 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.877493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.880320 (+ 2827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.880337 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.882652 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.882658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.882885 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.882970 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 20:57:09.883078 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:57:09.883275 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2534,"mutex_wait_us":7,"thread_start_us":2388,"threads_started":1}
I20260926 20:57:09.884109 29021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.870811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:39435 (local address 127.0.0.1:57532)
0926 20:57:09.872738 (+ 1927us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:09.872777 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:09.872963 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:09.875237 (+ 2274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:09.875243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:09.875267 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:09.875784 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.875798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.877734 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.877748 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.880747 (+ 2999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.880764 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.883011 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.883018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.883201 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.883231 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 20:57:09.883309 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 20:57:09.883435 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1685,"mutex_wait_us":771,"thread_start_us":1525,"threads_started":1}
I20260926 20:57:09.884388 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.870758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.8.249.130:39435)
0926 20:57:09.874356 (+ 3598us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:09.874367 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:09.874395 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:57:09.874495 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:09.874505 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:09.874747 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:09.874979 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:09.876016 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.877533 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.880943 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.882413 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.882562 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.882773 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:09.883843 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0926 20:57:09.884028 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3422,"thread_start_us":4006,"threads_started":1}
I20260926 20:57:09.884696 29022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.871410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.8.249.129:36499)
0926 20:57:09.872296 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:09.872309 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:09.873764 (+ 1455us) server_negotiation.cc:407] Connection header received
0926 20:57:09.873901 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:09.873909 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:09.874204 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:09.874420 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:09.875501 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.877297 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.880525 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.882413 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.883164 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.883425 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:09.883537 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 20:57:09.883723 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":417,"threads_started":1}
I20260926 20:57:09.888370 28675 tablet_service.cc:1467] Processing CreateTablet for tablet bd43b822f55f472382f783aa717b7fdd (DEFAULT_TABLE table=test-table [id=542e444f7d31430aa2fdd6b6132ee66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:09.888376 28809 tablet_service.cc:1467] Processing CreateTablet for tablet bd43b822f55f472382f783aa717b7fdd (DEFAULT_TABLE table=test-table [id=542e444f7d31430aa2fdd6b6132ee66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:09.889107 29023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.874184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:40071 (local address 127.0.0.1:35750)
0926 20:57:09.878830 (+ 4646us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:09.878850 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:09.878989 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:09.881551 (+ 2562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:09.881556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:09.881585 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:09.882057 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.882069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.884667 (+ 2598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.884675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.886659 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.886672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.888206 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.888212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.888400 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.888445 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:57:09.888581 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 20:57:09.888724 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4464,"thread_start_us":4310,"threads_started":1}
I20260926 20:57:09.889168 29024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.874485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35750 (local address 127.8.249.131:40071)
0926 20:57:09.881005 (+ 6520us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:09.881012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:09.881043 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:57:09.881134 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:09.881140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:09.881336 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:09.881575 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:09.882276 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.884480 (+ 2204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.886892 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.887971 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.888145 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.888399 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:09.888623 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 20:57:09.888798 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6331,"thread_start_us":6149,"threads_started":1}
I20260926 20:57:09.889905 28809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd43b822f55f472382f783aa717b7fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:09.889920 28675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd43b822f55f472382f783aa717b7fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:09.892853 28942 tablet_service.cc:1467] Processing CreateTablet for tablet bd43b822f55f472382f783aa717b7fdd (DEFAULT_TABLE table=test-table [id=542e444f7d31430aa2fdd6b6132ee66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:09.894112 28942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd43b822f55f472382f783aa717b7fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:09.906647 29026 tablet_bootstrap.cc:492] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234: Bootstrap starting.
I20260926 20:57:09.908025 29027 tablet_bootstrap.cc:492] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b: Bootstrap starting.
I20260926 20:57:09.909865 29026 tablet_bootstrap.cc:654] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:09.911006 29026 log.cc:824] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:09.911343 29027 tablet_bootstrap.cc:654] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:09.911358 29025 tablet_bootstrap.cc:492] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d: Bootstrap starting.
I20260926 20:57:09.912660 29027 log.cc:824] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:09.914467 29026 tablet_bootstrap.cc:492] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234: No bootstrap required, opened a new log
I20260926 20:57:09.914506 29025 tablet_bootstrap.cc:654] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:09.914762 29026 ts_tablet_manager.cc:1397] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 20:57:09.915731 29025 log.cc:824] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:09.917305 29027 tablet_bootstrap.cc:492] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b: No bootstrap required, opened a new log
I20260926 20:57:09.917693 29027 ts_tablet_manager.cc:1397] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 20:57:09.919071 29025 tablet_bootstrap.cc:492] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d: No bootstrap required, opened a new log
I20260926 20:57:09.919392 29025 ts_tablet_manager.cc:1397] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260926 20:57:09.925532 29026 raft_consensus.cc:348] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39435 } } peers { permanent_uuid: "bd202ce78a8f4336b8a0a81481f6ed1d" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36499 } } peers { permanent_uuid: "19227c15190945cd9ebc5dd8e8ee528b" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40071 } }
I20260926 20:57:09.926127 29026 raft_consensus.cc:374] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:09.926270 29026 raft_consensus.cc:729] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9454db3d34c4a3cb6b67bd3912cc234, State: Initialized, Role: FOLLOWER
I20260926 20:57:09.927141 29026 consensus_queue.cc:260] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39435 } } peers { permanent_uuid: "bd202ce78a8f4336b8a0a81481f6ed1d" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36499 } } peers { permanent_uuid: "19227c15190945cd9ebc5dd8e8ee528b" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40071 } }
I20260926 20:57:09.928545 28874 heartbeater.cc:503] Master 127.8.249.190:36715 was elected leader, sending a full tablet report...
I20260926 20:57:09.928234 29027 raft_consensus.cc:348] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39435 } } peers { permanent_uuid: "bd202ce78a8f4336b8a0a81481f6ed1d" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36499 } } peers { permanent_uuid: "19227c15190945cd9ebc5dd8e8ee528b" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40071 } }
I20260926 20:57:09.928694 29027 raft_consensus.cc:374] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:09.928800 29027 raft_consensus.cc:729] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19227c15190945cd9ebc5dd8e8ee528b, State: Initialized, Role: FOLLOWER
I20260926 20:57:09.929169 29026 ts_tablet_manager.cc:1428] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260926 20:57:09.929492 29027 consensus_queue.cc:260] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39435 } } peers { permanent_uuid: "bd202ce78a8f4336b8a0a81481f6ed1d" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36499 } } peers { permanent_uuid: "19227c15190945cd9ebc5dd8e8ee528b" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40071 } }
I20260926 20:57:09.930092 29025 raft_consensus.cc:348] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39435 } } peers { permanent_uuid: "bd202ce78a8f4336b8a0a81481f6ed1d" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36499 } } peers { permanent_uuid: "19227c15190945cd9ebc5dd8e8ee528b" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40071 } }
I20260926 20:57:09.930540 29025 raft_consensus.cc:374] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:09.930661 29025 raft_consensus.cc:729] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd202ce78a8f4336b8a0a81481f6ed1d, State: Initialized, Role: FOLLOWER
I20260926 20:57:09.931313 29027 ts_tablet_manager.cc:1428] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 20:57:09.931221 29025 consensus_queue.cc:260] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39435 } } peers { permanent_uuid: "bd202ce78a8f4336b8a0a81481f6ed1d" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36499 } } peers { permanent_uuid: "19227c15190945cd9ebc5dd8e8ee528b" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40071 } }
I20260926 20:57:09.931985 29007 heartbeater.cc:503] Master 127.8.249.190:36715 was elected leader, sending a full tablet report...
I20260926 20:57:09.932883 29025 ts_tablet_manager.cc:1428] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 20:57:09.978103 29031 raft_consensus.cc:482] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:57:09.978444 29031 raft_consensus.cc:504] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39435 } } peers { permanent_uuid: "bd202ce78a8f4336b8a0a81481f6ed1d" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36499 } } peers { permanent_uuid: "19227c15190945cd9ebc5dd8e8ee528b" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40071 } }
I20260926 20:57:09.980134 29031 leader_election.cc:288] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd202ce78a8f4336b8a0a81481f6ed1d (127.8.249.129:36499), 19227c15190945cd9ebc5dd8e8ee528b (127.8.249.131:40071)
I20260926 20:57:09.989746 29024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.980560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:51609 (local address 127.8.249.131:40071)
0926 20:57:09.980771 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:09.980777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:09.981540 (+ 763us) server_negotiation.cc:407] Connection header received
0926 20:57:09.981736 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:09.981741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:09.981820 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:09.981923 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:09.982821 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.983979 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.987314 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.988554 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.988723 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.988800 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:09.989359 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 20:57:09.989492 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:57:09.989748 29022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.980563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:54983 (local address 127.8.249.129:36499)
0926 20:57:09.980773 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:09.980779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:09.981540 (+ 761us) server_negotiation.cc:407] Connection header received
0926 20:57:09.981736 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:09.981741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:09.981820 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:09.981924 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:09.982821 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.983979 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.987314 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.988554 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.988723 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.988800 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:09.989359 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 20:57:09.989492 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:57:09.990087 29034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.980441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:40071 (local address 127.8.249.130:51609)
0926 20:57:09.981364 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:09.981390 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:09.981585 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:09.981969 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:09.981974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:09.982012 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:09.982626 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.982640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.984130 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.984137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.987119 (+ 2982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.987137 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.988862 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.988867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.989128 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.989208 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 20:57:09.989316 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:57:09.989499 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":325,"spinlock_wait_cycles":31104,"thread_start_us":524,"threads_started":1}
I20260926 20:57:09.990196 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:09.980516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:36499 (local address 127.8.249.130:54983)
0926 20:57:09.981425 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:09.981442 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:09.981585 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:09.981969 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:09.981974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:09.982009 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:09.982626 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.982640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.984133 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.984137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.987119 (+ 2982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:09.987137 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:09.988857 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:09.988865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:09.989131 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:09.989221 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 20:57:09.989316 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:57:09.989499 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":353,"spinlock_wait_cycles":8192,"thread_start_us":502,"threads_started":1}
I20260926 20:57:09.991183 28695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd43b822f55f472382f783aa717b7fdd" candidate_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd202ce78a8f4336b8a0a81481f6ed1d" is_pre_election: true
I20260926 20:57:09.991185 28962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd43b822f55f472382f783aa717b7fdd" candidate_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19227c15190945cd9ebc5dd8e8ee528b" is_pre_election: true
I20260926 20:57:09.991813 28962 raft_consensus.cc:2454] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9454db3d34c4a3cb6b67bd3912cc234 in term 0.
I20260926 20:57:09.991814 28695 raft_consensus.cc:2454] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9454db3d34c4a3cb6b67bd3912cc234 in term 0.
I20260926 20:57:09.992566 28765 leader_election.cc:302] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19227c15190945cd9ebc5dd8e8ee528b, b9454db3d34c4a3cb6b67bd3912cc234, bd202ce78a8f4336b8a0a81481f6ed1d; no voters:
I20260926 20:57:09.993055 29031 raft_consensus.cc:2781] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:57:09.993203 29031 raft_consensus.cc:482] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:57:09.993316 29031 raft_consensus.cc:3037] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:09.995733 29031 raft_consensus.cc:504] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39435 } } peers { permanent_uuid: "bd202ce78a8f4336b8a0a81481f6ed1d" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36499 } } peers { permanent_uuid: "19227c15190945cd9ebc5dd8e8ee528b" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40071 } }
I20260926 20:57:09.996356 29031 leader_election.cc:288] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [CANDIDATE]: Term 1 election: Requested vote from peers bd202ce78a8f4336b8a0a81481f6ed1d (127.8.249.129:36499), 19227c15190945cd9ebc5dd8e8ee528b (127.8.249.131:40071)
I20260926 20:57:09.996876 28695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd43b822f55f472382f783aa717b7fdd" candidate_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd202ce78a8f4336b8a0a81481f6ed1d"
I20260926 20:57:09.997081 28695 raft_consensus.cc:3037] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:09.999058 28962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd43b822f55f472382f783aa717b7fdd" candidate_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19227c15190945cd9ebc5dd8e8ee528b"
I20260926 20:57:09.999282 28962 raft_consensus.cc:3037] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:09.999607 28695 raft_consensus.cc:2454] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9454db3d34c4a3cb6b67bd3912cc234 in term 1.
I20260926 20:57:10.000080 28762 leader_election.cc:302] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9454db3d34c4a3cb6b67bd3912cc234, bd202ce78a8f4336b8a0a81481f6ed1d; no voters:
I20260926 20:57:10.000416 29031 raft_consensus.cc:2781] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:10.000895 29031 raft_consensus.cc:686] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [term 1 LEADER]: Becoming Leader. State: Replica: b9454db3d34c4a3cb6b67bd3912cc234, State: Running, Role: LEADER
I20260926 20:57:10.001459 29031 consensus_queue.cc:237] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39435 } } peers { permanent_uuid: "bd202ce78a8f4336b8a0a81481f6ed1d" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36499 } } peers { permanent_uuid: "19227c15190945cd9ebc5dd8e8ee528b" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40071 } }
I20260926 20:57:10.002562 28962 raft_consensus.cc:2454] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9454db3d34c4a3cb6b67bd3912cc234 in term 1.
I20260926 20:57:10.006469 28550 catalog_manager.cc:5495] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9454db3d34c4a3cb6b67bd3912cc234 (127.8.249.130). New cstate: current_term: 1 leader_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9454db3d34c4a3cb6b67bd3912cc234" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 39435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd202ce78a8f4336b8a0a81481f6ed1d" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19227c15190945cd9ebc5dd8e8ee528b" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40071 } health_report { overall_health: UNKNOWN } } }
W20260926 20:57:10.023123 28741 tablet.cc:2369] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:57:10.035259 28875 tablet.cc:2369] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:57:10.041692 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:10.036937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.8.249.130:39435)
0926 20:57:10.037149 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:10.037153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:10.037280 (+ 127us) server_negotiation.cc:407] Connection header received
0926 20:57:10.037441 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:10.037446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:10.037529 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:10.037649 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:10.037933 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:10.038607 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:10.039877 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:10.040270 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:10.040363 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:10.040770 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:10.041245 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:10.041354 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:10.041420 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 20:57:10.041482 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 20:57:10.044605 29008 tablet.cc:2369] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:57:10.049849 28962 raft_consensus.cc:1261] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b [term 1 FOLLOWER]: Refusing update from remote peer b9454db3d34c4a3cb6b67bd3912cc234: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:57:10.049947 28695 raft_consensus.cc:1261] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d [term 1 FOLLOWER]: Refusing update from remote peer b9454db3d34c4a3cb6b67bd3912cc234: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:57:10.050851 29036 consensus_queue.cc:1035] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd202ce78a8f4336b8a0a81481f6ed1d" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:57:10.051215 29031 consensus_queue.cc:1035] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19227c15190945cd9ebc5dd8e8ee528b" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 40071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:57:10.076813 28549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:57:10.077179 28549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:57:10.081245 28549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d528ae8abd8f4dc2975ca4d09c6ff4a2: Sending DeleteTablet for 3 replicas of tablet bd43b822f55f472382f783aa717b7fdd
I20260926 20:57:10.082430 28809 tablet_service.cc:1514] Processing DeleteTablet for tablet bd43b822f55f472382f783aa717b7fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:10 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:57532
I20260926 20:57:10.082440 28675 tablet_service.cc:1514] Processing DeleteTablet for tablet bd43b822f55f472382f783aa717b7fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:10 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:40866
I20260926 20:57:10.083096 28942 tablet_service.cc:1514] Processing DeleteTablet for tablet bd43b822f55f472382f783aa717b7fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:10 UTC) from {username='kudu', principal='kudu/127.8.249.190@KRBTEST.COM'} at 127.0.0.1:35750
I20260926 20:57:10.084065 29050 tablet_replica.cc:330] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b: stopping tablet replica
I20260926 20:57:10.084815 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 28612
I20260926 20:57:10.084741 29050 raft_consensus.cc:2229] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:57:10.085371 29050 raft_consensus.cc:2258] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:57:10.088699 29050 ts_tablet_manager.cc:1905] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:10.098125 29050 ts_tablet_manager.cc:1918] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:57:10.098363 29050 log.cc:1196] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/bd43b822f55f472382f783aa717b7fdd
I20260926 20:57:10.103547 29051 tablet_replica.cc:330] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234: stopping tablet replica
I20260926 20:57:10.104141 29051 raft_consensus.cc:2229] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:57:10.105167 29051 raft_consensus.cc:2258] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:57:10.107658 29051 ts_tablet_manager.cc:1905] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:10.111243 29050 ts_tablet_manager.cc:1939] T bd43b822f55f472382f783aa717b7fdd P 19227c15190945cd9ebc5dd8e8ee528b: Deleting consensus metadata
I20260926 20:57:10.111433 29049 tablet_replica.cc:330] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d: stopping tablet replica
I20260926 20:57:10.111972 29049 raft_consensus.cc:2229] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:57:10.112582 29049 raft_consensus.cc:2258] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 20:57:10 dist-test-slave-2dg3 krb5kdc[28506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.129@KRBTEST.COM
I20260926 20:57:10.113452 28538 catalog_manager.cc:4841] TS 19227c15190945cd9ebc5dd8e8ee528b (127.8.249.131:40071): tablet bd43b822f55f472382f783aa717b7fdd (table test-table [id=542e444f7d31430aa2fdd6b6132ee66b]) successfully deleted
I20260926 20:57:10.117038 29049 ts_tablet_manager.cc:1905] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:10.117398 29051 ts_tablet_manager.cc:1918] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:57:10.117655 29051 log.cc:1196] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/bd43b822f55f472382f783aa717b7fdd
I20260926 20:57:10.118284 29051 ts_tablet_manager.cc:1939] T bd43b822f55f472382f783aa717b7fdd P b9454db3d34c4a3cb6b67bd3912cc234: Deleting consensus metadata
I20260926 20:57:10.120169 28536 catalog_manager.cc:4841] TS b9454db3d34c4a3cb6b67bd3912cc234 (127.8.249.130:39435): tablet bd43b822f55f472382f783aa717b7fdd (table test-table [id=542e444f7d31430aa2fdd6b6132ee66b]) successfully deleted
I20260926 20:57:10.122498 29022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:10.086084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40876 (local address 127.8.249.129:36499)
0926 20:57:10.086320 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:10.086339 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:10.099471 (+ 13132us) server_negotiation.cc:407] Connection header received
0926 20:57:10.099816 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:10.099825 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:10.099955 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:10.100048 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:10.102356 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:10.104019 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:10.106675 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:10.107675 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:10.107810 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:10.114482 (+ 6672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:10.114495 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:10.114512 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:10.114558 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:10.118627 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:10.119588 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:10.119598 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:10.119601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:10.119782 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:10.120235 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:10.120241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:10.120245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:10.120703 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:10.120991 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:10.121815 (+ 824us) server_negotiation.cc:299] Negotiation successful
0926 20:57:10.122074 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:57:10.124300 29049 ts_tablet_manager.cc:1918] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:57:10.124665 29049 log.cc:1196] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790455916331309-9190-0/minicluster-data/ts-0/wal/wals/bd43b822f55f472382f783aa717b7fdd
I20260926 20:57:10.125133 29049 ts_tablet_manager.cc:1939] T bd43b822f55f472382f783aa717b7fdd P bd202ce78a8f4336b8a0a81481f6ed1d: Deleting consensus metadata
I20260926 20:57:10.126847 28735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:10.126746 28535 catalog_manager.cc:4841] TS bd202ce78a8f4336b8a0a81481f6ed1d (127.8.249.129:36499): tablet bd43b822f55f472382f783aa717b7fdd (table test-table [id=542e444f7d31430aa2fdd6b6132ee66b]) successfully deleted
I20260926 20:57:10.233381 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 28612
I20260926 20:57:10.251487 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 28746
Sep 26 20:57:10 dist-test-slave-2dg3 krb5kdc[28506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.130@KRBTEST.COM
I20260926 20:57:10.267702 29019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:10.252384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.8.249.130:39435)
0926 20:57:10.252575 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:10.252582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:10.252605 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:57:10.252677 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:10.252681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:10.252740 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:10.252866 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:10.254185 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:10.255168 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:10.256444 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:10.256883 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:10.257020 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:10.261463 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:10.261477 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:10.261497 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:10.261537 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:10.264797 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:10.265535 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:10.265543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:10.265545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:10.265679 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:10.266053 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:10.266059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:10.266062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:10.266415 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:10.266744 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:10.267026 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 20:57:10.267268 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:57:10.268359 28869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:10.395383 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 28746
I20260926 20:57:10.413689 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 28879
Sep 26 20:57:10 dist-test-slave-2dg3 krb5kdc[28506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.131@KRBTEST.COM
I20260926 20:57:10.428596 29024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:10.414460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.8.249.131:40071)
0926 20:57:10.414623 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:10.414627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:10.414644 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:57:10.414780 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:10.414784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:10.414908 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:10.414994 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:10.416286 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:10.416994 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:10.418193 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:10.418562 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:10.418650 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:10.423104 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:10.423118 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:10.423134 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:10.423172 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:10.425992 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:10.426628 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:10.426635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:10.426637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:10.426766 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:10.427378 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:10.427381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:10.427383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:10.427728 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:10.427952 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:10.428186 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 20:57:10.428352 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:57:10.429227 29002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:10.517733 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 28879
I20260926 20:57:10.534756 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 28519
I20260926 20:57:10.535984 28581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:10.648882 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 28519
2026-09-26T20:57:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4338 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T20:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:10Z Disabled control of system clock
I20260926 20:57:10.727921 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:39395
--webserver_interface=127.8.249.190
--webserver_port=0
--builtin_ntp_servers=127.8.249.148:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:39395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:57:11.060365 29060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:11.060667 29060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:11.060750 29060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:11.068156 29060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:11.068286 29060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:11.068324 29060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:11.068375 29060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:11.068419 29060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:11.077910 29060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:39395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:39395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:11.079193 29060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:11.080897 29060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:11.092563 29066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:11.092564 29065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:11.092583 29068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:11.093850 29060 server_base.cc:1034] running on GCE node
I20260926 20:57:11.094753 29060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:11.096326 29060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:11.097558 29060 hybrid_clock.cc:648] HybridClock initialized: now 1790456231097506 us; error 48 us; skew 500 ppm
I20260926 20:57:11.097929 29060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:57:11.103472 29060 webserver.cc:460] Webserver started at http://127.8.249.190:37537/ using document root <none> and password file <none>
I20260926 20:57:11.104209 29060 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:11.104350 29060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:11.104707 29060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:11.108232 29060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "17ffab57418d4f039395e9b20ecbc32c"
format_stamp: "Formatted at 2026-09-26 20:57:11 on dist-test-slave-2dg3"
server_key: "693a1cb8ad44e3b60bf311aa42d39414"
server_key_iv: "a08a2389aeed400dafe2a2b2888b18ab"
server_key_version: "encryptionkey@0"
I20260926 20:57:11.109017 29060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "17ffab57418d4f039395e9b20ecbc32c"
format_stamp: "Formatted at 2026-09-26 20:57:11 on dist-test-slave-2dg3"
server_key: "693a1cb8ad44e3b60bf311aa42d39414"
server_key_iv: "a08a2389aeed400dafe2a2b2888b18ab"
server_key_version: "encryptionkey@0"
I20260926 20:57:11.114221 29060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 20:57:11.117857 29074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:11.119370 29060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:11.119520 29060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "17ffab57418d4f039395e9b20ecbc32c"
format_stamp: "Formatted at 2026-09-26 20:57:11 on dist-test-slave-2dg3"
server_key: "693a1cb8ad44e3b60bf311aa42d39414"
server_key_iv: "a08a2389aeed400dafe2a2b2888b18ab"
server_key_version: "encryptionkey@0"
I20260926 20:57:11.119701 29060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:11.135309 29060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:11.136089 29060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:11.136329 29060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:11.155045 29060 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:39395
I20260926 20:57:11.155069 29125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:39395 every 8 connection(s)
I20260926 20:57:11.156486 29060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:11.160239 29126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:11.163223 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 29060
I20260926 20:57:11.163434 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:11.163841 9190 external_mini_cluster.cc:1428] Setting key 43103692876ec99c21d93b8068f9be3e
I20260926 20:57:11.173995 29126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c: Bootstrap starting.
I20260926 20:57:11.178061 29126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:11.179458 29126 log.cc:824] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:11.180250 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:11.165863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38090 (local address 127.8.249.190:39395)
0926 20:57:11.166990 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:11.167006 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:11.167060 (+ 54us) server_negotiation.cc:407] Connection header received
0926 20:57:11.169361 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:11.169398 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:11.170070 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:11.170555 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:11.171165 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:11.173142 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:11.174666 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:11.175789 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:11.175987 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:11.177163 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:11.177174 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:11.177191 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:11.177883 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:11.178026 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:11.178061 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:11.178196 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:11.178461 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 20:57:11.178861 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":347,"threads_started":1}
I20260926 20:57:11.182893 29126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c: No bootstrap required, opened a new log
I20260926 20:57:11.190472 29126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ffab57418d4f039395e9b20ecbc32c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 39395 } }
I20260926 20:57:11.190944 29126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:11.191082 29126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ffab57418d4f039395e9b20ecbc32c, State: Initialized, Role: FOLLOWER
I20260926 20:57:11.191815 29126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ffab57418d4f039395e9b20ecbc32c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 39395 } }
I20260926 20:57:11.192111 29126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:11.192293 29126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:11.192476 29126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:11.194676 29126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ffab57418d4f039395e9b20ecbc32c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 39395 } }
I20260926 20:57:11.195252 29126 leader_election.cc:302] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17ffab57418d4f039395e9b20ecbc32c; no voters:
I20260926 20:57:11.195868 29126 leader_election.cc:288] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:11.196116 29131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:11.196991 29131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [term 1 LEADER]: Becoming Leader. State: Replica: 17ffab57418d4f039395e9b20ecbc32c, State: Running, Role: LEADER
I20260926 20:57:11.197690 29131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ffab57418d4f039395e9b20ecbc32c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 39395 } }
I20260926 20:57:11.198014 29126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:11.200903 29133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17ffab57418d4f039395e9b20ecbc32c. Latest consensus state: current_term: 1 leader_uuid: "17ffab57418d4f039395e9b20ecbc32c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ffab57418d4f039395e9b20ecbc32c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 39395 } } }
I20260926 20:57:11.201344 29133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:11.201016 29132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17ffab57418d4f039395e9b20ecbc32c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ffab57418d4f039395e9b20ecbc32c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 39395 } } }
I20260926 20:57:11.201606 29132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:11.207427 29137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:11.213353 29137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:11.224395 29137 catalog_manager.cc:1347] Generated new cluster ID: 42536ec14e614cd49f127693c8dda0ec
I20260926 20:57:11.224537 29137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:11.240736 29137 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:11.242137 29137 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:11.249276 29137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c: Generated new TSK 0
I20260926 20:57:11.250171 29137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:11.257718 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.129:0
--local_ip_for_outbound_sockets=127.8.249.129
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:39395
--builtin_ntp_servers=127.8.249.148:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:57:11.602125 29150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:11.602407 29150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:11.602494 29150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:11.609931 29150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:11.610025 29150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:11.610173 29150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129
I20260926 20:57:11.619786 29150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.249.129
--webserver_port=0
--tserver_master_addrs=127.8.249.190:39395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.129
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:11.621011 29150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:11.622752 29150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:11.636384 29156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:11.636456 29158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:11.636464 29155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:11.637395 29150 server_base.cc:1034] running on GCE node
I20260926 20:57:11.638053 29150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:11.639081 29150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:11.640280 29150 hybrid_clock.cc:648] HybridClock initialized: now 1790456231640235 us; error 37 us; skew 500 ppm
I20260926 20:57:11.640600 29150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:57:11.642885 29150 webserver.cc:460] Webserver started at http://127.8.249.129:39445/ using document root <none> and password file <none>
I20260926 20:57:11.643600 29150 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:11.643723 29150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:11.644049 29150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:11.646994 29150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/data/instance:
uuid: "e93c6051dfdc482887149a418c50c2e8"
format_stamp: "Formatted at 2026-09-26 20:57:11 on dist-test-slave-2dg3"
server_key: "e20264e29363b76fca92706033d16d1c"
server_key_iv: "d24ef1cdb687e34e02c4a696301b036d"
server_key_version: "encryptionkey@0"
I20260926 20:57:11.647795 29150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance:
uuid: "e93c6051dfdc482887149a418c50c2e8"
format_stamp: "Formatted at 2026-09-26 20:57:11 on dist-test-slave-2dg3"
server_key: "e20264e29363b76fca92706033d16d1c"
server_key_iv: "d24ef1cdb687e34e02c4a696301b036d"
server_key_version: "encryptionkey@0"
I20260926 20:57:11.653409 29150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:57:11.657003 29164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:11.658341 29150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:57:11.658519 29150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/wal
uuid: "e93c6051dfdc482887149a418c50c2e8"
format_stamp: "Formatted at 2026-09-26 20:57:11 on dist-test-slave-2dg3"
server_key: "e20264e29363b76fca92706033d16d1c"
server_key_iv: "d24ef1cdb687e34e02c4a696301b036d"
server_key_version: "encryptionkey@0"
I20260926 20:57:11.658746 29150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:11.674297 29150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:11.675276 29150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:11.675595 29150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:11.676748 29150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:11.678244 29150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:11.678369 29150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:11.678500 29150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:11.678587 29150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:11.711958 29150 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.129:38971
I20260926 20:57:11.712006 29276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.129:38971 every 8 connection(s)
I20260926 20:57:11.713274 29150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/data/info.pb
I20260926 20:57:11.717805 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 29150
I20260926 20:57:11.718000 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-0/wal/instance
I20260926 20:57:11.718472 9190 external_mini_cluster.cc:1428] Setting key c8284ec8b9499d45e0b85a4a19fb4736
I20260926 20:57:11.721446 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.130:0
--local_ip_for_outbound_sockets=127.8.249.130
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:39395
--builtin_ntp_servers=127.8.249.148:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:57:11.726783 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:11.716032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.129:51971 (local address 127.8.249.190:39395)
0926 20:57:11.716497 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:11.716501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:11.717646 (+ 1145us) server_negotiation.cc:407] Connection header received
0926 20:57:11.719765 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:11.719771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:11.719867 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:11.719989 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:11.721230 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:11.722018 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:11.723910 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:11.724333 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:11.724469 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:11.725942 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:11.725946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:11.725949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:11.725968 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:11.726008 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:11.726020 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:11.726116 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:11.726347 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 20:57:11.726444 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":254,"threads_started":1}
I20260926 20:57:11.727838 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:11.716596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:39395 (local address 127.8.249.129:51971)
0926 20:57:11.717495 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:11.717534 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:11.719497 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:11.720235 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:11.720244 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:11.720385 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:11.721062 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:11.721080 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:11.722185 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:11.722189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:11.723757 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:11.723769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:11.724561 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:11.724570 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:11.724807 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:11.725588 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:57:11.725591 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:11.725632 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:57:11.725711 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:11.726115 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:11.726129 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:11.726139 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:57:11.726330 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 20:57:11.726737 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":290,"threads_started":1}
I20260926 20:57:11.729458 29277 heartbeater.cc:347] Connected to a master server at 127.8.249.190:39395
I20260926 20:57:11.729822 29277 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:11.730850 29277 heartbeater.cc:511] Master 127.8.249.190:39395 requested a full tablet report, sending...
I20260926 20:57:11.733349 29091 ts_manager.cc:194] Registered new tserver with Master: e93c6051dfdc482887149a418c50c2e8 (127.8.249.129:38971)
I20260926 20:57:11.735651 29091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.249.129:51971
W20260926 20:57:12.057010 29281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:12.057307 29281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:12.057404 29281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:12.064952 29281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:12.065063 29281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:12.065172 29281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130
I20260926 20:57:12.074620 29281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.249.130
--webserver_port=0
--tserver_master_addrs=127.8.249.190:39395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.130
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:12.075913 29281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:12.077709 29281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:12.091053 29286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:12.090951 29287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:12.090984 29289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:12.091856 29281 server_base.cc:1034] running on GCE node
I20260926 20:57:12.092486 29281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:12.093446 29281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:12.094676 29281 hybrid_clock.cc:648] HybridClock initialized: now 1790456232094628 us; error 57 us; skew 500 ppm
I20260926 20:57:12.095017 29281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:57:12.097313 29281 webserver.cc:460] Webserver started at http://127.8.249.130:37619/ using document root <none> and password file <none>
I20260926 20:57:12.098048 29281 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:12.098191 29281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:12.098567 29281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:12.101691 29281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/data/instance:
uuid: "9a9b591c86e24cf68f5f3f96eb84e346"
format_stamp: "Formatted at 2026-09-26 20:57:12 on dist-test-slave-2dg3"
server_key: "32f936e8d276c20a7fbb1fa6a978d055"
server_key_iv: "27e1a93a96cbb0af88c4839418edfe1c"
server_key_version: "encryptionkey@0"
I20260926 20:57:12.102543 29281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance:
uuid: "9a9b591c86e24cf68f5f3f96eb84e346"
format_stamp: "Formatted at 2026-09-26 20:57:12 on dist-test-slave-2dg3"
server_key: "32f936e8d276c20a7fbb1fa6a978d055"
server_key_iv: "27e1a93a96cbb0af88c4839418edfe1c"
server_key_version: "encryptionkey@0"
I20260926 20:57:12.107928 29281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 20:57:12.111907 29295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:12.113327 29281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:57:12.113507 29281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/wal
uuid: "9a9b591c86e24cf68f5f3f96eb84e346"
format_stamp: "Formatted at 2026-09-26 20:57:12 on dist-test-slave-2dg3"
server_key: "32f936e8d276c20a7fbb1fa6a978d055"
server_key_iv: "27e1a93a96cbb0af88c4839418edfe1c"
server_key_version: "encryptionkey@0"
I20260926 20:57:12.113736 29281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:12.140588 29281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:12.141467 29281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:12.141770 29281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:12.142828 29281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:12.144335 29281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:12.144457 29281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:12.144598 29281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:12.144690 29281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:12.181068 29281 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.130:45863
I20260926 20:57:12.181113 29407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.130:45863 every 8 connection(s)
I20260926 20:57:12.182377 29281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/data/info.pb
I20260926 20:57:12.191527 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 29281
I20260926 20:57:12.191748 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/wal/instance
I20260926 20:57:12.192231 9190 external_mini_cluster.cc:1428] Setting key 18d31cc2f85ce8205591358c8352fa7f
I20260926 20:57:12.195344 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.249.131:0
--local_ip_for_outbound_sockets=127.8.249.131
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:39395
--builtin_ntp_servers=127.8.249.148:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:57:12.195972 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.185417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.130:49981 (local address 127.8.249.190:39395)
0926 20:57:12.185609 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:12.185614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:12.187241 (+ 1627us) server_negotiation.cc:407] Connection header received
0926 20:57:12.188969 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:12.188974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:12.189052 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:12.189172 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:12.190149 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.191022 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.192725 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.193258 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.193480 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.194982 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:12.194989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:12.194992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:12.195013 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:12.195049 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:12.195063 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:12.195208 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:12.195470 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 20:57:12.195588 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:57:12.197145 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.186063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:39395 (local address 127.8.249.130:49981)
0926 20:57:12.187069 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:12.187111 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:12.188696 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:12.189344 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:12.189352 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:12.189443 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:12.189998 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.190014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.191179 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.191182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.192570 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.192582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.193508 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.193513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.193705 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.194544 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:57:12.194548 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:12.194593 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:57:12.194696 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:12.195207 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:12.195222 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:12.195234 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 20:57:12.195459 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 20:57:12.195922 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":287,"threads_started":1}
I20260926 20:57:12.198729 29408 heartbeater.cc:347] Connected to a master server at 127.8.249.190:39395
I20260926 20:57:12.199122 29408 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:12.200007 29408 heartbeater.cc:511] Master 127.8.249.190:39395 requested a full tablet report, sending...
I20260926 20:57:12.201682 29091 ts_manager.cc:194] Registered new tserver with Master: 9a9b591c86e24cf68f5f3f96eb84e346 (127.8.249.130:45863)
I20260926 20:57:12.202615 29091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.249.130:49981
W20260926 20:57:12.523598 29411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:12.523919 29411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:12.524083 29411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:12.531549 29411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:12.531721 29411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:12.531893 29411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131
I20260926 20:57:12.541923 29411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.249.131
--webserver_port=0
--tserver_master_addrs=127.8.249.190:39395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.249.131
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:12.543371 29411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:12.545188 29411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:12.558324 29416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:12.558271 29419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:12.558362 29417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:12.559417 29411 server_base.cc:1034] running on GCE node
I20260926 20:57:12.560110 29411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:12.561105 29411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:12.562345 29411 hybrid_clock.cc:648] HybridClock initialized: now 1790456232562300 us; error 59 us; skew 500 ppm
I20260926 20:57:12.562669 29411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 20:57:12.565006 29411 webserver.cc:460] Webserver started at http://127.8.249.131:45557/ using document root <none> and password file <none>
I20260926 20:57:12.565703 29411 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:12.565838 29411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:12.566166 29411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:12.569070 29411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/data/instance:
uuid: "4d3b30fcb9004875908d6d415082aeb7"
format_stamp: "Formatted at 2026-09-26 20:57:12 on dist-test-slave-2dg3"
server_key: "be58d8b03b42b7e52599889cd383cf14"
server_key_iv: "4113151d67fcecd987b4e09359b34d8d"
server_key_version: "encryptionkey@0"
I20260926 20:57:12.569869 29411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance:
uuid: "4d3b30fcb9004875908d6d415082aeb7"
format_stamp: "Formatted at 2026-09-26 20:57:12 on dist-test-slave-2dg3"
server_key: "be58d8b03b42b7e52599889cd383cf14"
server_key_iv: "4113151d67fcecd987b4e09359b34d8d"
server_key_version: "encryptionkey@0"
I20260926 20:57:12.574962 29411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 20:57:12.578315 29425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:12.579638 29411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:12.579818 29411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/wal
uuid: "4d3b30fcb9004875908d6d415082aeb7"
format_stamp: "Formatted at 2026-09-26 20:57:12 on dist-test-slave-2dg3"
server_key: "be58d8b03b42b7e52599889cd383cf14"
server_key_iv: "4113151d67fcecd987b4e09359b34d8d"
server_key_version: "encryptionkey@0"
I20260926 20:57:12.580034 29411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:12.595045 29411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:12.595858 29411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:12.596113 29411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:12.596990 29411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:57:12.598429 29411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:57:12.598517 29411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:12.598651 29411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:57:12.598737 29411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:12.630712 29411 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.131:37531
I20260926 20:57:12.630771 29537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.131:37531 every 8 connection(s)
I20260926 20:57:12.632300 29411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/data/info.pb
I20260926 20:57:12.633630 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 29411
I20260926 20:57:12.633837 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/wal/instance
I20260926 20:57:12.634398 9190 external_mini_cluster.cc:1428] Setting key 9472f29a11689dcf0fb3a2b6f9a9e53e
I20260926 20:57:12.645728 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.635511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:51073 (local address 127.8.249.190:39395)
0926 20:57:12.635728 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:12.635737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:12.637721 (+ 1984us) server_negotiation.cc:407] Connection header received
0926 20:57:12.639575 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:12.639579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:12.639646 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:12.639726 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:12.640772 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.641511 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.643142 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.643562 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.643665 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.645012 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:12.645016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:12.645019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:12.645037 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:12.645069 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:12.645080 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:12.645153 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:12.645390 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 20:57:12.645482 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:57:12.646785 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.636330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:39395 (local address 127.8.249.131:51073)
0926 20:57:12.637521 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:12.637569 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:12.639300 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:12.639935 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:12.639943 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:12.640032 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:12.640616 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.640632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.641661 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.641664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.642987 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.643000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.643750 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.643754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.643956 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.644705 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:57:12.644709 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:12.644751 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:57:12.644832 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:12.645172 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:12.645180 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:12.645187 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:57:12.645366 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 20:57:12.645677 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":321,"threads_started":1}
I20260926 20:57:12.648481 29538 heartbeater.cc:347] Connected to a master server at 127.8.249.190:39395
I20260926 20:57:12.648861 29538 heartbeater.cc:464] Registering TS with master...
I20260926 20:57:12.649772 29538 heartbeater.cc:511] Master 127.8.249.190:39395 requested a full tablet report, sending...
I20260926 20:57:12.651204 29091 ts_manager.cc:194] Registered new tserver with Master: 4d3b30fcb9004875908d6d415082aeb7 (127.8.249.131:37531)
I20260926 20:57:12.652016 29091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.249.131:51073
I20260926 20:57:12.662298 9190 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:57:12.671761 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.666364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.8.249.190:39395)
0926 20:57:12.666597 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:12.666605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:12.666910 (+ 305us) server_negotiation.cc:407] Connection header received
0926 20:57:12.667059 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:12.667062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:12.667136 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:12.667221 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:12.667653 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.668320 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.669596 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.669974 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.670070 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.671103 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:12.671107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:12.671110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:12.671126 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:12.671155 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:12.671165 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:12.671233 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:12.671355 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 20:57:12.671432 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:57:12.675441 29091 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:57:12.682287 29091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:57:12.718811 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.695056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.131:37531 (local address 127.0.0.1:35480)
0926 20:57:12.707513 (+ 12457us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:12.707532 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:12.707678 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:12.708665 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:12.708672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:12.708698 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:12.709118 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.709128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.712795 (+ 3667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.712800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.715825 (+ 3025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.715843 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.717792 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.717799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.718003 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.718058 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 20:57:12.718164 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 20:57:12.718390 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12335,"mutex_wait_us":155,"thread_start_us":11199,"threads_started":1}
I20260926 20:57:12.719240 29551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.694719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37164)
0926 20:57:12.706638 (+ 11919us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:12.706697 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:12.706912 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:12.708520 (+ 1608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:12.708526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:12.708563 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:12.709092 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.709110 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.712505 (+ 3395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.712511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.715817 (+ 3306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.715832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.718241 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.718250 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.718481 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.718521 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 20:57:12.718593 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:57:12.718743 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11644,"mutex_wait_us":438,"thread_start_us":12512,"threads_started":1}
I20260926 20:57:12.720057 29553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.695865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.8.249.131:37531)
0926 20:57:12.706217 (+ 10352us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:12.706229 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:12.707868 (+ 1639us) server_negotiation.cc:407] Connection header received
0926 20:57:12.708021 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:12.708034 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:12.708435 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:12.708708 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:12.709379 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.711066 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.716078 (+ 5012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.717578 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.719104 (+ 1526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.719427 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:12.719506 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 20:57:12.719670 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10127,"thread_start_us":9947,"threads_started":1}
I20260926 20:57:12.721122 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.695057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.8.249.129:38971)
0926 20:57:12.707872 (+ 12815us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:12.707887 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:12.707913 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:57:12.707997 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:12.708003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:12.708254 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:12.710516 (+ 2262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:12.710763 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.712309 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.716047 (+ 3738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.717949 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.720009 (+ 2060us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.720424 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:12.720505 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:57:12.720679 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12673,"thread_start_us":12557,"threads_started":1}
I20260926 20:57:12.724530 29473 tablet_service.cc:1467] Processing CreateTablet for tablet 3344d4f9381043088ee9b08190825d3d (DEFAULT_TABLE table=test-table [id=c92f20c84a5c4199a68c00d01e6b5386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:12.724606 29212 tablet_service.cc:1467] Processing CreateTablet for tablet 3344d4f9381043088ee9b08190825d3d (DEFAULT_TABLE table=test-table [id=c92f20c84a5c4199a68c00d01e6b5386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:12.725600 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.709311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:45863 (local address 127.0.0.1:51366)
0926 20:57:12.713802 (+ 4491us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:12.713829 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:12.715613 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:12.716733 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:12.716739 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:12.716768 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:12.717218 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.717236 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.721040 (+ 3804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.721045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.722712 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.722730 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.724833 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.724841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.725004 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.725038 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:57:12.725132 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:57:12.725270 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4314,"thread_start_us":4157,"threads_started":1}
I20260926 20:57:12.726171 29212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3344d4f9381043088ee9b08190825d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:12.726436 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.707485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.8.249.130:45863)
0926 20:57:12.712562 (+ 5077us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:12.712575 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:12.714874 (+ 2299us) server_negotiation.cc:407] Connection header received
0926 20:57:12.715900 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:12.715914 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:12.716220 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:12.716428 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:12.717495 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.720863 (+ 3368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.723045 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.724610 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.724788 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.725035 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:12.725846 (+ 811us) server_negotiation.cc:299] Negotiation successful
0926 20:57:12.726023 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4892,"thread_start_us":4768,"threads_started":1}
I20260926 20:57:12.726817 29473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3344d4f9381043088ee9b08190825d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:12.730106 29343 tablet_service.cc:1467] Processing CreateTablet for tablet 3344d4f9381043088ee9b08190825d3d (DEFAULT_TABLE table=test-table [id=c92f20c84a5c4199a68c00d01e6b5386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:57:12.731573 29343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3344d4f9381043088ee9b08190825d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:12.738857 29556 tablet_bootstrap.cc:492] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7: Bootstrap starting.
I20260926 20:57:12.739531 29277 heartbeater.cc:503] Master 127.8.249.190:39395 was elected leader, sending a full tablet report...
I20260926 20:57:12.743249 29557 tablet_bootstrap.cc:492] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8: Bootstrap starting.
I20260926 20:57:12.744537 29556 tablet_bootstrap.cc:654] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:12.745739 29558 tablet_bootstrap.cc:492] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346: Bootstrap starting.
I20260926 20:57:12.745904 29556 log.cc:824] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:12.747118 29557 tablet_bootstrap.cc:654] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:12.748380 29557 log.cc:824] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:12.749150 29558 tablet_bootstrap.cc:654] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:12.749840 29556 tablet_bootstrap.cc:492] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7: No bootstrap required, opened a new log
I20260926 20:57:12.750169 29556 ts_tablet_manager.cc:1397] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260926 20:57:12.750321 29558 log.cc:824] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:12.751735 29557 tablet_bootstrap.cc:492] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8: No bootstrap required, opened a new log
I20260926 20:57:12.752019 29557 ts_tablet_manager.cc:1397] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 20:57:12.752986 29558 tablet_bootstrap.cc:492] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346: No bootstrap required, opened a new log
I20260926 20:57:12.753233 29558 ts_tablet_manager.cc:1397] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 20:57:12.760293 29558 raft_consensus.cc:348] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93c6051dfdc482887149a418c50c2e8" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 38971 } } peers { permanent_uuid: "4d3b30fcb9004875908d6d415082aeb7" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37531 } } peers { permanent_uuid: "9a9b591c86e24cf68f5f3f96eb84e346" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45863 } }
I20260926 20:57:12.760703 29558 raft_consensus.cc:374] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:12.760848 29558 raft_consensus.cc:729] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9b591c86e24cf68f5f3f96eb84e346, State: Initialized, Role: FOLLOWER
I20260926 20:57:12.761505 29558 consensus_queue.cc:260] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93c6051dfdc482887149a418c50c2e8" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 38971 } } peers { permanent_uuid: "4d3b30fcb9004875908d6d415082aeb7" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37531 } } peers { permanent_uuid: "9a9b591c86e24cf68f5f3f96eb84e346" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45863 } }
I20260926 20:57:12.761632 29556 raft_consensus.cc:348] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93c6051dfdc482887149a418c50c2e8" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 38971 } } peers { permanent_uuid: "4d3b30fcb9004875908d6d415082aeb7" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37531 } } peers { permanent_uuid: "9a9b591c86e24cf68f5f3f96eb84e346" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45863 } }
I20260926 20:57:12.762276 29556 raft_consensus.cc:374] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:12.762429 29556 raft_consensus.cc:729] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3b30fcb9004875908d6d415082aeb7, State: Initialized, Role: FOLLOWER
I20260926 20:57:12.762718 29408 heartbeater.cc:503] Master 127.8.249.190:39395 was elected leader, sending a full tablet report...
I20260926 20:57:12.763269 29556 consensus_queue.cc:260] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93c6051dfdc482887149a418c50c2e8" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 38971 } } peers { permanent_uuid: "4d3b30fcb9004875908d6d415082aeb7" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37531 } } peers { permanent_uuid: "9a9b591c86e24cf68f5f3f96eb84e346" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45863 } }
I20260926 20:57:12.764071 29558 ts_tablet_manager.cc:1428] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 20:57:12.765103 29538 heartbeater.cc:503] Master 127.8.249.190:39395 was elected leader, sending a full tablet report...
I20260926 20:57:12.765738 29556 ts_tablet_manager.cc:1428] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7: Time spent starting tablet: real 0.015s user 0.014s sys 0.003s
I20260926 20:57:12.766659 29557 raft_consensus.cc:348] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93c6051dfdc482887149a418c50c2e8" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 38971 } } peers { permanent_uuid: "4d3b30fcb9004875908d6d415082aeb7" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37531 } } peers { permanent_uuid: "9a9b591c86e24cf68f5f3f96eb84e346" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45863 } }
I20260926 20:57:12.767756 29557 raft_consensus.cc:374] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:12.767904 29557 raft_consensus.cc:729] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93c6051dfdc482887149a418c50c2e8, State: Initialized, Role: FOLLOWER
I20260926 20:57:12.768535 29557 consensus_queue.cc:260] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93c6051dfdc482887149a418c50c2e8" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 38971 } } peers { permanent_uuid: "4d3b30fcb9004875908d6d415082aeb7" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37531 } } peers { permanent_uuid: "9a9b591c86e24cf68f5f3f96eb84e346" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45863 } }
I20260926 20:57:12.770452 29557 ts_tablet_manager.cc:1428] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
I20260926 20:57:12.852816 29563 raft_consensus.cc:482] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:57:12.853116 29563 raft_consensus.cc:504] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93c6051dfdc482887149a418c50c2e8" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 38971 } } peers { permanent_uuid: "4d3b30fcb9004875908d6d415082aeb7" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37531 } } peers { permanent_uuid: "9a9b591c86e24cf68f5f3f96eb84e346" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45863 } }
I20260926 20:57:12.854835 29563 leader_election.cc:288] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e93c6051dfdc482887149a418c50c2e8 (127.8.249.129:38971), 9a9b591c86e24cf68f5f3f96eb84e346 (127.8.249.130:45863)
I20260926 20:57:12.861354 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.855092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.8.249.131:47431)
0926 20:57:12.855384 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:12.855404 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:12.855551 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:12.855927 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:12.855932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:12.855958 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:12.856179 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.856193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.857418 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.857426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.859530 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.859544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.860810 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.860813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.860923 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.860947 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:57:12.861017 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:57:12.861106 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260926 20:57:12.861454 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.855190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:47431 (local address 127.8.249.129:38971)
0926 20:57:12.855386 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:12.855393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:12.855505 (+ 112us) server_negotiation.cc:407] Connection header received
0926 20:57:12.855701 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:12.855705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:12.855777 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:12.855881 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:12.856377 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.857293 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.859674 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.860671 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.860835 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.860905 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:12.861065 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 20:57:12.861193 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:57:12.862216 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.855350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.249.131:37835 (local address 127.8.249.130:45863)
0926 20:57:12.855567 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:12.855572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:12.856164 (+ 592us) server_negotiation.cc:407] Connection header received
0926 20:57:12.856287 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:12.856302 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:12.856388 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:12.856486 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:57:12.857057 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.858008 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.860170 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.861344 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.861506 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.861571 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:12.861825 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 20:57:12.861955 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 20:57:12.862277 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.855248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.130:45863 (local address 127.8.249.131:37835)
0926 20:57:12.856055 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:12.856069 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:12.856181 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:12.856501 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:12.856504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:12.856525 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:57:12.856898 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.856909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.858179 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.858184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.860007 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:12.860016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.861518 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.861526 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:12.861675 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.861706 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 20:57:12.861794 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 20:57:12.861910 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":30,"thread_start_us":438,"threads_started":1}
I20260926 20:57:12.862744 29232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3344d4f9381043088ee9b08190825d3d" candidate_uuid: "4d3b30fcb9004875908d6d415082aeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93c6051dfdc482887149a418c50c2e8" is_pre_election: true
I20260926 20:57:12.863410 29232 raft_consensus.cc:2454] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3b30fcb9004875908d6d415082aeb7 in term 0.
I20260926 20:57:12.863502 29363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3344d4f9381043088ee9b08190825d3d" candidate_uuid: "4d3b30fcb9004875908d6d415082aeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9b591c86e24cf68f5f3f96eb84e346" is_pre_election: true
I20260926 20:57:12.864060 29427 leader_election.cc:302] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3b30fcb9004875908d6d415082aeb7, e93c6051dfdc482887149a418c50c2e8; no voters:
I20260926 20:57:12.864071 29363 raft_consensus.cc:2454] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3b30fcb9004875908d6d415082aeb7 in term 0.
I20260926 20:57:12.864398 29563 raft_consensus.cc:2781] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:57:12.864540 29563 raft_consensus.cc:482] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:57:12.864646 29563 raft_consensus.cc:3037] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:12.866850 29563 raft_consensus.cc:504] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93c6051dfdc482887149a418c50c2e8" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 38971 } } peers { permanent_uuid: "4d3b30fcb9004875908d6d415082aeb7" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37531 } } peers { permanent_uuid: "9a9b591c86e24cf68f5f3f96eb84e346" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45863 } }
I20260926 20:57:12.867416 29563 leader_election.cc:288] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [CANDIDATE]: Term 1 election: Requested vote from peers e93c6051dfdc482887149a418c50c2e8 (127.8.249.129:38971), 9a9b591c86e24cf68f5f3f96eb84e346 (127.8.249.130:45863)
I20260926 20:57:12.867841 29232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3344d4f9381043088ee9b08190825d3d" candidate_uuid: "4d3b30fcb9004875908d6d415082aeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93c6051dfdc482887149a418c50c2e8"
I20260926 20:57:12.867904 29363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3344d4f9381043088ee9b08190825d3d" candidate_uuid: "4d3b30fcb9004875908d6d415082aeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9b591c86e24cf68f5f3f96eb84e346"
I20260926 20:57:12.868021 29232 raft_consensus.cc:3037] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:12.868108 29363 raft_consensus.cc:3037] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:12.870064 29232 raft_consensus.cc:2454] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3b30fcb9004875908d6d415082aeb7 in term 1.
I20260926 20:57:12.870297 29363 raft_consensus.cc:2454] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3b30fcb9004875908d6d415082aeb7 in term 1.
I20260926 20:57:12.870395 29427 leader_election.cc:302] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3b30fcb9004875908d6d415082aeb7, e93c6051dfdc482887149a418c50c2e8; no voters:
I20260926 20:57:12.870673 29563 raft_consensus.cc:2781] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:12.871140 29563 raft_consensus.cc:686] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3b30fcb9004875908d6d415082aeb7, State: Running, Role: LEADER
I20260926 20:57:12.871649 29563 consensus_queue.cc:237] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93c6051dfdc482887149a418c50c2e8" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 38971 } } peers { permanent_uuid: "4d3b30fcb9004875908d6d415082aeb7" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37531 } } peers { permanent_uuid: "9a9b591c86e24cf68f5f3f96eb84e346" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45863 } }
I20260926 20:57:12.875931 29090 catalog_manager.cc:5495] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3b30fcb9004875908d6d415082aeb7 (127.8.249.131). New cstate: current_term: 1 leader_uuid: "4d3b30fcb9004875908d6d415082aeb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e93c6051dfdc482887149a418c50c2e8" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 38971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3b30fcb9004875908d6d415082aeb7" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a9b591c86e24cf68f5f3f96eb84e346" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45863 } health_report { overall_health: UNKNOWN } } }
W20260926 20:57:12.884392 29539 tablet.cc:2369] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:57:12.906477 29553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.901299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35490 (local address 127.8.249.131:37531)
0926 20:57:12.901510 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:12.901515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:12.901533 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:57:12.901726 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:12.901730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:12.901812 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:12.901963 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:12.902297 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.903092 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.904427 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.904825 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.904957 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.905466 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:12.905945 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:12.906109 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:12.906160 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 20:57:12.906231 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:57:12.914711 29363 raft_consensus.cc:1261] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346 [term 1 FOLLOWER]: Refusing update from remote peer 4d3b30fcb9004875908d6d415082aeb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:57:12.914739 29232 raft_consensus.cc:1261] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8 [term 1 FOLLOWER]: Refusing update from remote peer 4d3b30fcb9004875908d6d415082aeb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:57:12.915791 29566 consensus_queue.cc:1035] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e93c6051dfdc482887149a418c50c2e8" member_type: VOTER last_known_addr { host: "127.8.249.129" port: 38971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:57:12.916137 29563 consensus_queue.cc:1035] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9b591c86e24cf68f5f3f96eb84e346" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:57:12.927400 29569 mvcc.cc:205] Tried to move back new op lower bound from 7333708730009235456 to 7333708729846046720. Current Snapshot: MvccSnapshot[applied={T|T < 7333708730009235456}]
I20260926 20:57:12.933118 29570 mvcc.cc:205] Tried to move back new op lower bound from 7333708730009235456 to 7333708729846046720. Current Snapshot: MvccSnapshot[applied={T|T < 7333708730009235456}]
I20260926 20:57:12.949857 29091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:57:12.950183 29091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:57:12.954416 29091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17ffab57418d4f039395e9b20ecbc32c: Sending DeleteTablet for 3 replicas of tablet 3344d4f9381043088ee9b08190825d3d
I20260926 20:57:12.955847 29473 tablet_service.cc:1514] Processing DeleteTablet for tablet 3344d4f9381043088ee9b08190825d3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:12 UTC) from {username='slave'} at 127.0.0.1:35480
I20260926 20:57:12.955861 29211 tablet_service.cc:1514] Processing DeleteTablet for tablet 3344d4f9381043088ee9b08190825d3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:12 UTC) from {username='slave'} at 127.0.0.1:37164
I20260926 20:57:12.956300 29343 tablet_service.cc:1514] Processing DeleteTablet for tablet 3344d4f9381043088ee9b08190825d3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:12 UTC) from {username='slave'} at 127.0.0.1:51366
I20260926 20:57:12.957007 29212 tablet_service.cc:1514] Processing DeleteTablet for tablet 3344d4f9381043088ee9b08190825d3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:57:12 UTC) from {username='slave'} at 127.0.0.1:37164
I20260926 20:57:12.958985 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 29150
I20260926 20:57:12.962404 29581 tablet_replica.cc:330] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346: stopping tablet replica
I20260926 20:57:12.963127 29581 raft_consensus.cc:2229] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:57:12.963804 29581 raft_consensus.cc:2258] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:57:12.966426 29581 ts_tablet_manager.cc:1905] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:12.980197 29579 tablet_replica.cc:330] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7: stopping tablet replica
I20260926 20:57:12.980401 29580 tablet_replica.cc:330] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8: stopping tablet replica
I20260926 20:57:12.980686 29579 raft_consensus.cc:2229] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:57:12.980798 29580 raft_consensus.cc:2229] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:57:12.981281 29579 raft_consensus.cc:2258] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:57:12.981314 29580 raft_consensus.cc:2258] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:57:12.982937 29581 ts_tablet_manager.cc:1918] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:57:12.983193 29581 log.cc:1196] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-1/wal/wals/3344d4f9381043088ee9b08190825d3d
I20260926 20:57:12.983824 29581 ts_tablet_manager.cc:1939] T 3344d4f9381043088ee9b08190825d3d P 9a9b591c86e24cf68f5f3f96eb84e346: Deleting consensus metadata
I20260926 20:57:12.983827 29580 ts_tablet_manager.cc:1905] T 3344d4f9381043088ee9b08190825d3d P e93c6051dfdc482887149a418c50c2e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:12.984133 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:12.960514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.8.249.129:38971)
0926 20:57:12.960743 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:12.960753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:12.971440 (+ 10687us) server_negotiation.cc:407] Connection header received
0926 20:57:12.972000 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:12.972010 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:12.972145 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:12.972236 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:12.973080 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.974697 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.977255 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:12.978189 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:12.978311 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:12.982275 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:12.982285 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:12.982303 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:12.983137 (+ 834us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:12.983195 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:12.983228 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:12.983317 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:12.983566 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 20:57:12.983692 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 20:57:12.985530 29272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:12.985785 29077 catalog_manager.cc:4841] TS 9a9b591c86e24cf68f5f3f96eb84e346 (127.8.249.130:45863): tablet 3344d4f9381043088ee9b08190825d3d (table test-table [id=c92f20c84a5c4199a68c00d01e6b5386]) successfully deleted
I20260926 20:57:12.989028 29579 ts_tablet_manager.cc:1905] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:57:12.997092 29579 ts_tablet_manager.cc:1918] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:57:12.997300 29579 log.cc:1196] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7: Deleting WAL directory at /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790455916331309-9190-0/minicluster-data/ts-2/wal/wals/3344d4f9381043088ee9b08190825d3d
I20260926 20:57:12.997944 29579 ts_tablet_manager.cc:1939] T 3344d4f9381043088ee9b08190825d3d P 4d3b30fcb9004875908d6d415082aeb7: Deleting consensus metadata
I20260926 20:57:12.999981 29077 catalog_manager.cc:4841] TS 4d3b30fcb9004875908d6d415082aeb7 (127.8.249.131:37531): tablet 3344d4f9381043088ee9b08190825d3d (table test-table [id=c92f20c84a5c4199a68c00d01e6b5386]) successfully deleted
I20260926 20:57:13.076241 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 29150
W20260926 20:57:13.100632 29076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.249.129:38971 (error 108)
I20260926 20:57:13.100828 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 29281
I20260926 20:57:13.102945 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.102379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37184)
0926 20:57:13.102571 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.102719 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 20:57:13.103225 29076 catalog_manager.cc:4584] TS e93c6051dfdc482887149a418c50c2e8 (127.8.249.129:38971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3344d4f9381043088ee9b08190825d3d: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
I20260926 20:57:13.108304 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.102008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.8.249.130:45863)
0926 20:57:13.102295 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:13.102301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:13.102319 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:57:13.102382 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:13.102387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:13.102500 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:13.102614 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:13.103098 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:13.104002 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:13.105251 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:13.105655 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:13.105786 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:13.106896 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:13.106905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:13.106919 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:13.107492 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:13.107537 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:13.107556 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:13.107681 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:13.107804 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 20:57:13.107910 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260926 20:57:13.108965 29403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:13.148159 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.147582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37194)
0926 20:57:13.147837 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.147953 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 20:57:13.149307 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.148907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37206)
0926 20:57:13.149073 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.149163 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 20:57:13.156756 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.156235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37212)
0926 20:57:13.156486 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.156571 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 20:57:13.157940 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.157501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37220)
0926 20:57:13.157696 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.157784 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 20:57:13.198654 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.198105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37236)
0926 20:57:13.198333 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.198448 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 20:57:13.199898 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.199436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37248)
0926 20:57:13.199663 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.199763 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 20:57:13.216178 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 29281
I20260926 20:57:13.234469 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.234030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37264)
0926 20:57:13.234231 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.234326 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 20:57:13.235648 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.235274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37272)
0926 20:57:13.235444 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.235530 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 20:57:13.239202 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 29411
I20260926 20:57:13.246953 29553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.239928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.8.249.131:37531)
0926 20:57:13.240086 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:13.240090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:13.240141 (+ 51us) server_negotiation.cc:407] Connection header received
0926 20:57:13.240364 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:13.240369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:13.240462 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:13.240546 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:13.241053 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:13.241841 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:13.243723 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:13.244154 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:13.244243 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:13.245454 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:13.245462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:13.245478 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:13.246078 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:13.246123 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:13.246149 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:13.246229 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:13.246409 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 20:57:13.246488 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:57:13.247709 29533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:13.300966 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.300476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37280)
0926 20:57:13.300687 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.300783 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 20:57:13.302016 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.301650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37292)
0926 20:57:13.301827 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.301903 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 20:57:13.337611 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.337138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37294)
0926 20:57:13.337354 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.337447 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 20:57:13.338639 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.338299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:37296)
0926 20:57:13.338451 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.338517 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 20:57:13.368469 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 29411
I20260926 20:57:13.388101 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 29060
I20260926 20:57:13.389426 29121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:13.498463 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:13.497881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.129:38971 (local address 127.0.0.1:39832)
0926 20:57:13.498090 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:13.498227 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.129:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 20:57:13.498492 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 29060
2026-09-26T20:57:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2852 ms)
[----------] 6 tests from EncryptionPolicyTest (21554 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:13 dist-test-slave-2dg3 krb5kdc[29590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:13 dist-test-slave-2dg3 krb5kdc[29590](info): set up 2 sockets
Sep 26 20:57:13 dist-test-slave-2dg3 krb5kdc[29590](info): commencing operation
krb5kdc: starting...
W20260926 20:57:14.611416 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:14 dist-test-slave-2dg3 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:57:14.874568 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:15.221792 29603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:15.222214 29603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:15.222401 29603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:15.230182 29603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:15.230365 29603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:15.230466 29603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:15.230573 29603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:15.230648 29603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:15.243943 29603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33979
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:15.245373 29603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:15.246754 29603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:15.259033 29608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:15.258986 29609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:15.258942 29611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:15.259871 29603 server_base.cc:1034] running on GCE node
I20260926 20:57:15.260756 29603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:15.262293 29603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:15.263523 29603 hybrid_clock.cc:648] HybridClock initialized: now 1790456235263457 us; error 63 us; skew 500 ppm
I20260926 20:57:15.263878 29603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:15 dist-test-slave-2dg3 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:15.267094 29603 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:57:15.268597 29603 webserver.cc:460] Webserver started at http://127.0.0.1:46645/ using document root <none> and password file <none>
I20260926 20:57:15.269307 29603 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:15.269466 29603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:15.269760 29603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:15.272645 29603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "beacb874abab4b8e9c072ea3a7c7476e"
format_stamp: "Formatted at 2026-09-26 20:57:15 on dist-test-slave-2dg3"
server_key: "79924d4e449c0d56f0676a903464f0e8"
server_key_iv: "e0a620e0c9cbeacded5ae5f5743d5ecf"
server_key_version: "encryptionkey@0"
I20260926 20:57:15.273344 29603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "beacb874abab4b8e9c072ea3a7c7476e"
format_stamp: "Formatted at 2026-09-26 20:57:15 on dist-test-slave-2dg3"
server_key: "79924d4e449c0d56f0676a903464f0e8"
server_key_iv: "e0a620e0c9cbeacded5ae5f5743d5ecf"
server_key_version: "encryptionkey@0"
I20260926 20:57:15.278923 29603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 20:57:15.282656 29618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:15.284041 29603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 20:57:15.284240 29603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "beacb874abab4b8e9c072ea3a7c7476e"
format_stamp: "Formatted at 2026-09-26 20:57:15 on dist-test-slave-2dg3"
server_key: "79924d4e449c0d56f0676a903464f0e8"
server_key_iv: "e0a620e0c9cbeacded5ae5f5743d5ecf"
server_key_version: "encryptionkey@0"
I20260926 20:57:15.284451 29603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:15.298138 29603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:15.303867 29603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:15.304201 29603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:15.322525 29603 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33979
I20260926 20:57:15.322546 29669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33979 every 8 connection(s)
I20260926 20:57:15.324054 29603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:15.327981 29670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:15.330971 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 29603
I20260926 20:57:15.331203 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:15.331868 9190 external_mini_cluster.cc:1428] Setting key 53b867646eb6277cda4d40ba1e4edac2
I20260926 20:57:15.341693 29670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e: Bootstrap starting.
I20260926 20:57:15.345467 29670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:15.346534 29670 log.cc:824] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:57:15 dist-test-slave-2dg3 krb5kdc[29590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:57:15.349723 29670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e: No bootstrap required, opened a new log
I20260926 20:57:15.356947 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:15.334066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39423 (local address 127.0.0.1:33979)
0926 20:57:15.334963 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:15.334979 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:15.335037 (+ 58us) server_negotiation.cc:407] Connection header received
0926 20:57:15.337300 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:15.337341 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:15.337959 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:15.338462 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:15.339977 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:15.341951 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:15.343359 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:15.344451 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:15.344645 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:15.349092 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:15.349101 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:15.349118 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:15.349156 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:15.352363 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:15.353293 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:15.353300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:15.353306 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:15.353482 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:15.353988 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:15.353991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:15.353993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:15.354569 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:15.354990 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:15.355478 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 20:57:15.355823 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":369,"threads_started":1}
I20260926 20:57:15.357980 29670 raft_consensus.cc:348] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beacb874abab4b8e9c072ea3a7c7476e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33979 } }
I20260926 20:57:15.358405 29670 raft_consensus.cc:374] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:15.358538 29670 raft_consensus.cc:729] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beacb874abab4b8e9c072ea3a7c7476e, State: Initialized, Role: FOLLOWER
I20260926 20:57:15.359206 29670 consensus_queue.cc:260] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beacb874abab4b8e9c072ea3a7c7476e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33979 } }
I20260926 20:57:15.359483 29670 raft_consensus.cc:388] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:15.359648 29670 raft_consensus.cc:482] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:15.359856 29670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:15.361418 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:15.362820 29670 raft_consensus.cc:504] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beacb874abab4b8e9c072ea3a7c7476e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33979 } }
I20260926 20:57:15.363433 29670 leader_election.cc:302] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beacb874abab4b8e9c072ea3a7c7476e; no voters:
I20260926 20:57:15.364087 29670 leader_election.cc:288] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:15.364167 29675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:15.366919 29670 sys_catalog.cc:564] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:15.367091 29675 raft_consensus.cc:686] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [term 1 LEADER]: Becoming Leader. State: Replica: beacb874abab4b8e9c072ea3a7c7476e, State: Running, Role: LEADER
I20260926 20:57:15.369100 29675 consensus_queue.cc:237] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beacb874abab4b8e9c072ea3a7c7476e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33979 } }
I20260926 20:57:15.380561 29676 sys_catalog.cc:455] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beacb874abab4b8e9c072ea3a7c7476e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beacb874abab4b8e9c072ea3a7c7476e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33979 } } }
I20260926 20:57:15.380941 29676 sys_catalog.cc:458] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:15.381412 29677 sys_catalog.cc:455] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [sys.catalog]: SysCatalogTable state changed. Reason: New leader beacb874abab4b8e9c072ea3a7c7476e. Latest consensus state: current_term: 1 leader_uuid: "beacb874abab4b8e9c072ea3a7c7476e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beacb874abab4b8e9c072ea3a7c7476e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33979 } } }
I20260926 20:57:15.381647 29677 sys_catalog.cc:458] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:15.393973 29686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:15.399672 29686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:15.422570 29686 catalog_manager.cc:1347] Generated new cluster ID: b16634c755364dfb8aff6792091ce33b
I20260926 20:57:15.422704 29686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:15.437237 29686 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:15.439389 29686 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:15.462077 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:15.424847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37765 (local address 127.0.0.1:33979)
0926 20:57:15.425053 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:15.425061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:15.435807 (+ 10746us) server_negotiation.cc:407] Connection header received
0926 20:57:15.436425 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:15.436434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:15.436548 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:15.436648 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:15.439455 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:15.441571 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:15.444146 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:15.445160 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:15.445298 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:15.450555 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:15.450571 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:15.450580 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:15.450646 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:15.455892 (+ 5246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:15.457475 (+ 1583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:15.457485 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:15.457491 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:15.457745 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:15.458553 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:15.458561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:15.458567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:15.459125 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:15.460981 (+ 1856us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:15.461232 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 20:57:15.461519 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:57:15.464978 29686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beacb874abab4b8e9c072ea3a7c7476e: Generated new TSK 0
I20260926 20:57:15.466578 29686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:15.475389 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 29603
I20260926 20:57:15.477743 29665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:15.576839 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 29603
W20260926 20:57:15.594708 29588 connection.cc:535] client connection to 127.0.0.1:33979 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33979): Connection reset by peer (error 104)
2026-09-26T20:57:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2069 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:15 dist-test-slave-2dg3 krb5kdc[29704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:15 dist-test-slave-2dg3 krb5kdc[29704](info): set up 2 sockets
Sep 26 20:57:15 dist-test-slave-2dg3 krb5kdc[29704](info): commencing operation
krb5kdc: starting...
W20260926 20:57:16.679209 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:16 dist-test-slave-2dg3 krb5kdc[29704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:57:16.937443 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:17.273671 29717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:17.273986 29717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:17.274080 29717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:17.281641 29717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:17.281792 29717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:17.281864 29717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:17.281919 29717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:17.281963 29717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:17.292241 29717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45131
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:17.293524 29717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:17.294754 29717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:17.306682 29722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:17.306576 29723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:17.306646 29725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:17.307564 29717 server_base.cc:1034] running on GCE node
I20260926 20:57:17.308439 29717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:17.309881 29717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:17.311103 29717 hybrid_clock.cc:648] HybridClock initialized: now 1790456237311044 us; error 55 us; skew 500 ppm
I20260926 20:57:17.311444 29717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:17 dist-test-slave-2dg3 krb5kdc[29704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:17.319882 29717 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:57:17.321542 29717 webserver.cc:460] Webserver started at http://127.0.0.1:43785/ using document root <none> and password file <none>
I20260926 20:57:17.322238 29717 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:17.322340 29717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:17.322597 29717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:17.325695 29717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "d4f7b968fdb941cd97733eb7f2f6cb00"
format_stamp: "Formatted at 2026-09-26 20:57:17 on dist-test-slave-2dg3"
server_key: "59b78c2114fc3d711f0dbb38089227aa"
server_key_iv: "a07da85593d9fcb17ae8d431dee484c9"
server_key_version: "encryptionkey@0"
I20260926 20:57:17.326404 29717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "d4f7b968fdb941cd97733eb7f2f6cb00"
format_stamp: "Formatted at 2026-09-26 20:57:17 on dist-test-slave-2dg3"
server_key: "59b78c2114fc3d711f0dbb38089227aa"
server_key_iv: "a07da85593d9fcb17ae8d431dee484c9"
server_key_version: "encryptionkey@0"
I20260926 20:57:17.331614 29717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 20:57:17.335162 29732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:17.336637 29717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:57:17.336858 29717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "d4f7b968fdb941cd97733eb7f2f6cb00"
format_stamp: "Formatted at 2026-09-26 20:57:17 on dist-test-slave-2dg3"
server_key: "59b78c2114fc3d711f0dbb38089227aa"
server_key_iv: "a07da85593d9fcb17ae8d431dee484c9"
server_key_version: "encryptionkey@0"
I20260926 20:57:17.337064 29717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:17.351661 29717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:17.356137 29717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:17.356437 29717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:17.375053 29717 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45131
I20260926 20:57:17.375077 29783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45131 every 8 connection(s)
I20260926 20:57:17.376482 29717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:17.380246 29784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:17.382865 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 29717
I20260926 20:57:17.383061 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:17.383569 9190 external_mini_cluster.cc:1428] Setting key 739da60b3ed6175b3527911222b80d80
I20260926 20:57:17.392551 29784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00: Bootstrap starting.
I20260926 20:57:17.396262 29784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:17.397215 29787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:17.385452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36747 (local address 127.0.0.1:45131)
0926 20:57:17.386319 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:17.386331 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:17.386373 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:57:17.388101 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:17.388124 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:17.388526 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:17.388857 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:17.389442 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:17.390767 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:17.392726 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:17.393419 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:17.393555 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:17.394919 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:17.394928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:17.394942 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:17.395503 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:17.395584 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:17.395607 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:17.395722 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:17.395981 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 20:57:17.396177 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":387,"threads_started":1}
I20260926 20:57:17.397306 29784 log.cc:824] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:17.400635 29784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00: No bootstrap required, opened a new log
I20260926 20:57:17.401612 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:17.411233 29784 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f7b968fdb941cd97733eb7f2f6cb00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260926 20:57:17.411775 29784 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:17.411887 29784 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f7b968fdb941cd97733eb7f2f6cb00, State: Initialized, Role: FOLLOWER
I20260926 20:57:17.412745 29784 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f7b968fdb941cd97733eb7f2f6cb00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260926 20:57:17.413071 29784 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:17.413245 29784 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:17.413467 29784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:17.413851 29787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:17.406816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52119 (local address 127.0.0.1:45131)
0926 20:57:17.407019 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:17.407026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:17.407423 (+ 397us) server_negotiation.cc:407] Connection header received
0926 20:57:17.407578 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:17.407584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:17.407670 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:17.407793 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:17.408276 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:17.409455 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:17.410837 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:17.411486 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:17.411624 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:17.412932 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:17.412939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:17.412945 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:17.412974 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:17.413040 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:17.413069 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:17.413203 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:17.413434 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 20:57:17.413504 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:57:17.416468 29784 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f7b968fdb941cd97733eb7f2f6cb00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260926 20:57:17.417068 29784 leader_election.cc:302] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4f7b968fdb941cd97733eb7f2f6cb00; no voters:
I20260926 20:57:17.417759 29784 leader_election.cc:288] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:17.417893 29794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:17.418644 29794 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [term 1 LEADER]: Becoming Leader. State: Replica: d4f7b968fdb941cd97733eb7f2f6cb00, State: Running, Role: LEADER
I20260926 20:57:17.419243 29794 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f7b968fdb941cd97733eb7f2f6cb00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260926 20:57:17.419929 29784 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:17.422478 29796 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4f7b968fdb941cd97733eb7f2f6cb00. Latest consensus state: current_term: 1 leader_uuid: "d4f7b968fdb941cd97733eb7f2f6cb00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f7b968fdb941cd97733eb7f2f6cb00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } } }
I20260926 20:57:17.422981 29796 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:17.424145 29795 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4f7b968fdb941cd97733eb7f2f6cb00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f7b968fdb941cd97733eb7f2f6cb00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } } }
I20260926 20:57:17.425248 29795 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:17.427412 29801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:17.434142 29801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:17.442839 29801 catalog_manager.cc:1347] Generated new cluster ID: a0e32a726f7743bd8d811fea433b9050
I20260926 20:57:17.442965 29801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:17.464936 29801 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:17.466938 29801 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:17.486292 29801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4f7b968fdb941cd97733eb7f2f6cb00: Generated new TSK 0
I20260926 20:57:17.487448 29801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:17.495661 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 29717
W20260926 20:57:17.496986 29779 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36747
W20260926 20:57:17.497795 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 29717: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:57:17.497982 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 29717
W20260926 20:57:17.515105 29702 connection.cc:535] client connection to 127.0.0.1:45131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45131): Connection reset by peer (error 104)
2026-09-26T20:57:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:17 dist-test-slave-2dg3 krb5kdc[29815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:17 dist-test-slave-2dg3 krb5kdc[29815](info): set up 2 sockets
Sep 26 20:57:17 dist-test-slave-2dg3 krb5kdc[29815](info): commencing operation
krb5kdc: starting...
W20260926 20:57:18.611256 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.010s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:18 dist-test-slave-2dg3 krb5kdc[29815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:57:18.868196 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:19.198632 29828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:19.199021 29828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:19.199203 29828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:19.206497 29828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:19.206679 29828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:19.206801 29828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:19.206931 29828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:19.207019 29828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:19.219261 29828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37627
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:19.220767 29828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:19.222098 29828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:19.234320 29836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:19.234452 29833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:19.234421 29834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:19.235724 29828 server_base.cc:1034] running on GCE node
I20260926 20:57:19.236603 29828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:19.238152 29828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:19.239352 29828 hybrid_clock.cc:648] HybridClock initialized: now 1790456239239310 us; error 39 us; skew 500 ppm
I20260926 20:57:19.239705 29828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:19 dist-test-slave-2dg3 krb5kdc[29815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:19.243039 29828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:57:19.244711 29828 webserver.cc:460] Webserver started at http://127.0.0.1:37655/ using document root <none> and password file <none>
I20260926 20:57:19.245440 29828 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:19.245569 29828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:19.245903 29828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:19.249152 29828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "58a31333729a43b1b5599041c2aad6d0"
format_stamp: "Formatted at 2026-09-26 20:57:19 on dist-test-slave-2dg3"
server_key: "657f05f80325708d7f5a45c0bfdd7230"
server_key_iv: "1f468d51029ede27c045ebc62090b71f"
server_key_version: "encryptionkey@0"
I20260926 20:57:19.249953 29828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "58a31333729a43b1b5599041c2aad6d0"
format_stamp: "Formatted at 2026-09-26 20:57:19 on dist-test-slave-2dg3"
server_key: "657f05f80325708d7f5a45c0bfdd7230"
server_key_iv: "1f468d51029ede27c045ebc62090b71f"
server_key_version: "encryptionkey@0"
I20260926 20:57:19.255379 29828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:57:19.258991 29843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:19.260344 29828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:57:19.260529 29828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "58a31333729a43b1b5599041c2aad6d0"
format_stamp: "Formatted at 2026-09-26 20:57:19 on dist-test-slave-2dg3"
server_key: "657f05f80325708d7f5a45c0bfdd7230"
server_key_iv: "1f468d51029ede27c045ebc62090b71f"
server_key_version: "encryptionkey@0"
I20260926 20:57:19.260743 29828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:19.266649 29828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:19.271065 29828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:19.271368 29828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:19.289901 29828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37627
I20260926 20:57:19.289986 29894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37627 every 8 connection(s)
I20260926 20:57:19.291445 29828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:19.293795 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 29828
I20260926 20:57:19.293977 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:19.294406 9190 external_mini_cluster.cc:1428] Setting key 4f552fd2290f5aa755706fea95f7581a
I20260926 20:57:19.295696 29895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:19.302183 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:19.296234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46837 (local address 127.0.0.1:37627)
0926 20:57:19.297016 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:19.297027 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:19.297069 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:57:19.298644 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:19.298666 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:19.299103 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:19.299507 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:19.300555 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:19.300564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:19.300577 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:19.300590 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:19.300662 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:19.300688 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:19.300947 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:19.301093 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 20:57:19.301284 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":347,"threads_started":1}
I20260926 20:57:19.305894 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:19.308115 29895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0: Bootstrap starting.
I20260926 20:57:19.311939 29895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:19.312855 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:19.310399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32941 (local address 127.0.0.1:37627)
0926 20:57:19.310601 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:19.310606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:19.310918 (+ 312us) server_negotiation.cc:407] Connection header received
0926 20:57:19.311104 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:19.311108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:19.311155 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:19.311238 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:19.312168 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:19.312172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:19.312175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:19.312188 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:19.312232 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:19.312247 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:19.312339 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:19.312507 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 20:57:19.312549 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 20:57:19.313299 29895 log.cc:824] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:19.316205 29895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0: No bootstrap required, opened a new log
I20260926 20:57:19.324028 29895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a31333729a43b1b5599041c2aad6d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37627 } }
I20260926 20:57:19.324473 29895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:19.324586 29895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a31333729a43b1b5599041c2aad6d0, State: Initialized, Role: FOLLOWER
I20260926 20:57:19.325201 29895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a31333729a43b1b5599041c2aad6d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37627 } }
I20260926 20:57:19.325455 29895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:19.325606 29895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:19.325789 29895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:19.328310 29895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a31333729a43b1b5599041c2aad6d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37627 } }
I20260926 20:57:19.328838 29895 leader_election.cc:302] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58a31333729a43b1b5599041c2aad6d0; no voters:
I20260926 20:57:19.329439 29895 leader_election.cc:288] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:19.329583 29905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:19.330266 29905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [term 1 LEADER]: Becoming Leader. State: Replica: 58a31333729a43b1b5599041c2aad6d0, State: Running, Role: LEADER
I20260926 20:57:19.330899 29905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a31333729a43b1b5599041c2aad6d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37627 } }
I20260926 20:57:19.331686 29895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:19.333665 29906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58a31333729a43b1b5599041c2aad6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a31333729a43b1b5599041c2aad6d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37627 } } }
I20260926 20:57:19.334126 29906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:19.334933 29907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a31333729a43b1b5599041c2aad6d0. Latest consensus state: current_term: 1 leader_uuid: "58a31333729a43b1b5599041c2aad6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a31333729a43b1b5599041c2aad6d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37627 } } }
I20260926 20:57:19.335294 29907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:19.335140 29913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:19.341106 29913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:19.349885 29913 catalog_manager.cc:1347] Generated new cluster ID: 943d9857a3a3484083e0a685190f4c9d
I20260926 20:57:19.350010 29913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:19.365242 29913 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:19.365710 29913 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:19.376215 29913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58a31333729a43b1b5599041c2aad6d0: Generated new TSK 0
I20260926 20:57:19.377292 29913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:19.391465 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 29828
W20260926 20:57:19.392565 29890 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46837
W20260926 20:57:19.393261 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 29828: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:57:19.393431 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 29828
2026-09-26T20:57:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1894 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:19 dist-test-slave-2dg3 krb5kdc[29926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:19 dist-test-slave-2dg3 krb5kdc[29926](info): set up 2 sockets
Sep 26 20:57:19 dist-test-slave-2dg3 krb5kdc[29926](info): commencing operation
krb5kdc: starting...
W20260926 20:57:20.499296 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:20 dist-test-slave-2dg3 krb5kdc[29926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:57:20.758273 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42657
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:21.094524 29939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:21.094890 29939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:21.095041 29939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:21.102622 29939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:21.102726 29939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:21.102878 29939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:21.102949 29939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:21.102994 29939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:21.113036 29939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:21.114246 29939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:21.115563 29939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:21.127581 29944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:21.127660 29945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:21.127581 29947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:21.129171 29939 server_base.cc:1034] running on GCE node
I20260926 20:57:21.130038 29939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:21.131531 29939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:21.132722 29939 hybrid_clock.cc:648] HybridClock initialized: now 1790456241132662 us; error 56 us; skew 500 ppm
I20260926 20:57:21.133029 29939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:21 dist-test-slave-2dg3 krb5kdc[29926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:21.136125 29939 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:57:21.137610 29939 webserver.cc:460] Webserver started at http://127.0.0.1:35029/ using document root <none> and password file <none>
I20260926 20:57:21.138335 29939 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:21.138477 29939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:21.138840 29939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:21.141798 29939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "918678e1152a4b53b1e9d946357ee4d9"
format_stamp: "Formatted at 2026-09-26 20:57:21 on dist-test-slave-2dg3"
server_key: "61349af7020211473b9780a6f2005b67"
server_key_iv: "d4a00e54fb8ade4cc782912e550ae250"
server_key_version: "encryptionkey@0"
I20260926 20:57:21.142567 29939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "918678e1152a4b53b1e9d946357ee4d9"
format_stamp: "Formatted at 2026-09-26 20:57:21 on dist-test-slave-2dg3"
server_key: "61349af7020211473b9780a6f2005b67"
server_key_iv: "d4a00e54fb8ade4cc782912e550ae250"
server_key_version: "encryptionkey@0"
I20260926 20:57:21.147804 29939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 20:57:21.151358 29954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:21.152688 29939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:57:21.152864 29939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "918678e1152a4b53b1e9d946357ee4d9"
format_stamp: "Formatted at 2026-09-26 20:57:21 on dist-test-slave-2dg3"
server_key: "61349af7020211473b9780a6f2005b67"
server_key_iv: "d4a00e54fb8ade4cc782912e550ae250"
server_key_version: "encryptionkey@0"
I20260926 20:57:21.153127 29939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:21.174925 29939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:21.179543 29939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:21.179953 29939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:21.199086 29939 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42657
I20260926 20:57:21.199141 30005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42657 every 8 connection(s)
I20260926 20:57:21.200515 29939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:21.204038 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 29939
I20260926 20:57:21.204248 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:21.204492 30006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:21.204746 9190 external_mini_cluster.cc:1428] Setting key 4b1eb0dd28283b6d11bdaa8cd82a714d
I20260926 20:57:21.220986 30006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9: Bootstrap starting.
Sep 26 20:57:21 dist-test-slave-2dg3 krb5kdc[29926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:57:21.224812 30006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:21.225905 30006 log.cc:824] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:21.229283 30006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9: No bootstrap required, opened a new log
I20260926 20:57:21.230479 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:21.206490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33453 (local address 127.0.0.1:42657)
0926 20:57:21.207289 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:21.207301 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:21.207342 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:57:21.209904 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:21.209941 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:21.210562 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:21.211114 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:21.212907 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:21.215027 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:21.216890 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:21.218002 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:21.218206 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:21.222721 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:21.222730 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:21.222746 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:21.222895 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:21.226184 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:21.227103 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:21.227112 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:21.227121 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:21.227335 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:21.227774 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:21.227777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:21.227779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:21.228362 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:21.228569 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:21.229037 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 20:57:21.229488 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":347,"threads_started":1}
I20260926 20:57:21.234385 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:21.237659 30006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918678e1152a4b53b1e9d946357ee4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42657 } }
I20260926 20:57:21.238049 30006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:21.238174 30006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918678e1152a4b53b1e9d946357ee4d9, State: Initialized, Role: FOLLOWER
I20260926 20:57:21.238827 30006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918678e1152a4b53b1e9d946357ee4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42657 } }
I20260926 20:57:21.239187 30006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:21.239374 30006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:21.239570 30006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:21.242727 30006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918678e1152a4b53b1e9d946357ee4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42657 } }
I20260926 20:57:21.243399 30006 leader_election.cc:302] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 918678e1152a4b53b1e9d946357ee4d9; no voters:
I20260926 20:57:21.244028 30006 leader_election.cc:288] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:21.244128 30016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:21.246150 30016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [term 1 LEADER]: Becoming Leader. State: Replica: 918678e1152a4b53b1e9d946357ee4d9, State: Running, Role: LEADER
I20260926 20:57:21.246667 30006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:21.247189 30016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918678e1152a4b53b1e9d946357ee4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42657 } }
I20260926 20:57:21.251123 30017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "918678e1152a4b53b1e9d946357ee4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918678e1152a4b53b1e9d946357ee4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42657 } } }
I20260926 20:57:21.251577 30017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:21.251987 30018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 918678e1152a4b53b1e9d946357ee4d9. Latest consensus state: current_term: 1 leader_uuid: "918678e1152a4b53b1e9d946357ee4d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918678e1152a4b53b1e9d946357ee4d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42657 } } }
I20260926 20:57:21.252349 30018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:21.258311 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:21.239158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47089 (local address 127.0.0.1:42657)
0926 20:57:21.239365 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:21.239374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:21.239902 (+ 528us) server_negotiation.cc:407] Connection header received
0926 20:57:21.240081 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:21.240085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:21.240147 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:21.240246 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:21.242118 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:21.242863 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:21.244734 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:21.245686 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:21.245853 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:21.248829 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:21.248838 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:21.248846 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:21.248899 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:21.253343 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:21.254258 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:21.254266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:21.254272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:21.254440 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:21.255038 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:21.255047 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:21.255053 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:21.255550 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:21.255878 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:21.257492 (+ 1614us) server_negotiation.cc:299] Negotiation successful
0926 20:57:21.257781 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:57:21.261698 30022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:21.266672 30022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:21.275460 30022 catalog_manager.cc:1347] Generated new cluster ID: e8380120292342dd9622d261b96ecff0
I20260926 20:57:21.275571 30022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:21.290252 30022 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:21.292174 30022 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:21.301188 30022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 918678e1152a4b53b1e9d946357ee4d9: Generated new TSK 0
I20260926 20:57:21.302245 30022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:21.316118 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 29939
I20260926 20:57:21.317274 30001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:21.407164 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 29939
2026-09-26T20:57:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2013 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:21 dist-test-slave-2dg3 krb5kdc[30038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:21 dist-test-slave-2dg3 krb5kdc[30038](info): set up 2 sockets
Sep 26 20:57:21 dist-test-slave-2dg3 krb5kdc[30038](info): commencing operation
krb5kdc: starting...
W20260926 20:57:22.499328 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:22 dist-test-slave-2dg3 krb5kdc[30038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:57:22.758271 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:23.092954 30051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:23.093308 30051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:23.093489 30051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:23.100817 30051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:23.100987 30051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:23.101101 30051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:23.101213 30051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:23.101281 30051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:23.111479 30051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:23.112982 30051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:23.114497 30051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:23.126636 30056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:23.126670 30057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:23.126744 30059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:23.127384 30051 server_base.cc:1034] running on GCE node
I20260926 20:57:23.128199 30051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:23.129643 30051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:23.130895 30051 hybrid_clock.cc:648] HybridClock initialized: now 1790456243130852 us; error 59 us; skew 500 ppm
I20260926 20:57:23.131256 30051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:23 dist-test-slave-2dg3 krb5kdc[30038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:23.139680 30051 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:57:23.141268 30051 webserver.cc:460] Webserver started at http://127.0.0.1:36157/ using document root <none> and password file <none>
I20260926 20:57:23.142047 30051 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:23.142158 30051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:23.142508 30051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:23.145700 30051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "44d4b8a131aa4515b27ad542ddf93922"
format_stamp: "Formatted at 2026-09-26 20:57:23 on dist-test-slave-2dg3"
server_key: "eb99828a73ceb341b1235c247bce901d"
server_key_iv: "eb9040af7161476b82e9f9d5658bd39a"
server_key_version: "encryptionkey@0"
I20260926 20:57:23.146493 30051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "44d4b8a131aa4515b27ad542ddf93922"
format_stamp: "Formatted at 2026-09-26 20:57:23 on dist-test-slave-2dg3"
server_key: "eb99828a73ceb341b1235c247bce901d"
server_key_iv: "eb9040af7161476b82e9f9d5658bd39a"
server_key_version: "encryptionkey@0"
I20260926 20:57:23.151957 30051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 20:57:23.155833 30066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:23.157298 30051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:57:23.157492 30051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "44d4b8a131aa4515b27ad542ddf93922"
format_stamp: "Formatted at 2026-09-26 20:57:23 on dist-test-slave-2dg3"
server_key: "eb99828a73ceb341b1235c247bce901d"
server_key_iv: "eb9040af7161476b82e9f9d5658bd39a"
server_key_version: "encryptionkey@0"
I20260926 20:57:23.157749 30051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:23.185096 30051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:23.189452 30051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:23.189764 30051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:23.208698 30051 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41267
I20260926 20:57:23.208709 30117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41267 every 8 connection(s)
I20260926 20:57:23.210160 30051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:23.214890 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 30051
I20260926 20:57:23.215106 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:23.215503 30118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:23.215664 9190 external_mini_cluster.cc:1428] Setting key c1b3a8a059e4996b9b09760e51e4ba37
I20260926 20:57:23.231633 30118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922: Bootstrap starting.
I20260926 20:57:23.231822 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:23.217762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52565 (local address 127.0.0.1:41267)
0926 20:57:23.219715 (+ 1953us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:23.219730 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:23.219781 (+ 51us) server_negotiation.cc:407] Connection header received
0926 20:57:23.221363 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:23.221399 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:23.221955 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:23.222440 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:23.223056 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:23.225099 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:23.226474 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:23.227579 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:23.227793 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:23.228930 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:23.228940 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:23.228960 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:23.229716 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:23.229814 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:23.229845 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:23.229956 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:23.230160 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:57:23.230433 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1689,"thread_start_us":1461,"threads_started":1}
I20260926 20:57:23.235116 30118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:23.235538 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:23.236209 30118 log.cc:824] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:23.239449 30118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922: No bootstrap required, opened a new log
I20260926 20:57:23.247611 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:23.240327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60953 (local address 127.0.0.1:41267)
0926 20:57:23.240517 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:23.240524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:23.241086 (+ 562us) server_negotiation.cc:407] Connection header received
0926 20:57:23.241307 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:23.241310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:23.241372 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:23.241462 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:23.242158 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:23.243000 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:23.244955 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:23.245452 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:23.245547 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:23.246836 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:23.246849 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:23.246852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:23.246870 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:23.246908 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:23.246923 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:23.246997 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:23.247175 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 20:57:23.247257 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:57:23.250360 30118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d4b8a131aa4515b27ad542ddf93922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41267 } }
I20260926 20:57:23.250888 30118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:23.251020 30118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d4b8a131aa4515b27ad542ddf93922, State: Initialized, Role: FOLLOWER
I20260926 20:57:23.251668 30118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d4b8a131aa4515b27ad542ddf93922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41267 } }
I20260926 20:57:23.251974 30118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:23.252173 30118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:23.252372 30118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:23.254877 30118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d4b8a131aa4515b27ad542ddf93922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41267 } }
I20260926 20:57:23.255424 30118 leader_election.cc:302] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44d4b8a131aa4515b27ad542ddf93922; no voters:
I20260926 20:57:23.256117 30118 leader_election.cc:288] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:23.256219 30128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:23.257017 30128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [term 1 LEADER]: Becoming Leader. State: Replica: 44d4b8a131aa4515b27ad542ddf93922, State: Running, Role: LEADER
I20260926 20:57:23.257812 30128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d4b8a131aa4515b27ad542ddf93922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41267 } }
I20260926 20:57:23.258697 30118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:23.259773 30130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44d4b8a131aa4515b27ad542ddf93922. Latest consensus state: current_term: 1 leader_uuid: "44d4b8a131aa4515b27ad542ddf93922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d4b8a131aa4515b27ad542ddf93922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41267 } } }
I20260926 20:57:23.260227 30130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:23.260792 30129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44d4b8a131aa4515b27ad542ddf93922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d4b8a131aa4515b27ad542ddf93922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41267 } } }
I20260926 20:57:23.261282 30129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:23.266669 30133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:23.272003 30133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:23.281250 30133 catalog_manager.cc:1347] Generated new cluster ID: 34e8dc0d206047f7a52778bc8178b660
I20260926 20:57:23.281394 30133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:23.306954 30133 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:23.309033 30133 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:23.322342 30133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44d4b8a131aa4515b27ad542ddf93922: Generated new TSK 0
I20260926 20:57:23.323508 30133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:23.337833 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30051
W20260926 20:57:23.339006 30113 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52565
W20260926 20:57:23.339756 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30051: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:57:23.339913 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 30051
2026-09-26T20:57:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:23 dist-test-slave-2dg3 krb5kdc[30149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:23 dist-test-slave-2dg3 krb5kdc[30149](info): set up 2 sockets
Sep 26 20:57:23 dist-test-slave-2dg3 krb5kdc[30149](info): commencing operation
krb5kdc: starting...
W20260926 20:57:24.431393 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:24 dist-test-slave-2dg3 krb5kdc[30149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:57:24.697901 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32841
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:25.031513 30162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:25.031876 30162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:25.032042 30162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:25.039439 30162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:25.039633 30162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:25.039746 30162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:25.039855 30162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:25.039942 30162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:25.050227 30162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:25.051702 30162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:25.053145 30162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:25.065573 30167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:25.065474 30168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:25.065735 30170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:25.066772 30162 server_base.cc:1034] running on GCE node
I20260926 20:57:25.067663 30162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:25.069139 30162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:25.070380 30162 hybrid_clock.cc:648] HybridClock initialized: now 1790456245070333 us; error 41 us; skew 500 ppm
I20260926 20:57:25.070753 30162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:25 dist-test-slave-2dg3 krb5kdc[30149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:25.073891 30162 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:57:25.075562 30162 webserver.cc:460] Webserver started at http://127.0.0.1:45179/ using document root <none> and password file <none>
I20260926 20:57:25.076287 30162 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:25.076418 30162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:25.076736 30162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:25.079802 30162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "7ff9de5f90544c75afc2977c8bdc6df3"
format_stamp: "Formatted at 2026-09-26 20:57:25 on dist-test-slave-2dg3"
server_key: "568f7a3ae7a84f51b2a17882cd732978"
server_key_iv: "7f9c0473cae794c850fb7b59177dd8d6"
server_key_version: "encryptionkey@0"
I20260926 20:57:25.080749 30162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "7ff9de5f90544c75afc2977c8bdc6df3"
format_stamp: "Formatted at 2026-09-26 20:57:25 on dist-test-slave-2dg3"
server_key: "568f7a3ae7a84f51b2a17882cd732978"
server_key_iv: "7f9c0473cae794c850fb7b59177dd8d6"
server_key_version: "encryptionkey@0"
I20260926 20:57:25.085983 30162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:57:25.089709 30177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:25.091221 30162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 20:57:25.091411 30162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "7ff9de5f90544c75afc2977c8bdc6df3"
format_stamp: "Formatted at 2026-09-26 20:57:25 on dist-test-slave-2dg3"
server_key: "568f7a3ae7a84f51b2a17882cd732978"
server_key_iv: "7f9c0473cae794c850fb7b59177dd8d6"
server_key_version: "encryptionkey@0"
I20260926 20:57:25.091635 30162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:25.097281 30162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:25.101470 30162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:25.101783 30162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:25.120265 30162 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32841
I20260926 20:57:25.120307 30228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32841 every 8 connection(s)
I20260926 20:57:25.121698 30162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:25.123476 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 30162
I20260926 20:57:25.123687 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:25.124114 9190 external_mini_cluster.cc:1428] Setting key 7ca55010cd82657b988b52a8e7590352
I20260926 20:57:25.125964 30229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:25.132990 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:25.125919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52487 (local address 127.0.0.1:32841)
0926 20:57:25.126739 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:25.126753 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:25.126826 (+ 73us) server_negotiation.cc:407] Connection header received
0926 20:57:25.128970 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:25.128998 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:25.129484 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:25.129947 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:25.130971 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:25.130983 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:25.131008 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:25.131025 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:25.131111 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:25.131155 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:25.131384 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:25.131515 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 20:57:25.131714 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":364,"threads_started":1}
I20260926 20:57:25.137053 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:25.140669 30229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3: Bootstrap starting.
I20260926 20:57:25.144942 30229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:25.145153 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:25.142605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38663 (local address 127.0.0.1:32841)
0926 20:57:25.142869 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:25.142875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:25.143151 (+ 276us) server_negotiation.cc:407] Connection header received
0926 20:57:25.143298 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:25.143303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:25.143366 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:25.143483 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:25.144372 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:25.144378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:25.144382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:25.144400 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:25.144451 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:25.144472 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:25.144591 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:25.144697 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 20:57:25.144764 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:57:25.146341 30229 log.cc:824] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:25.149216 30229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3: No bootstrap required, opened a new log
I20260926 20:57:25.156798 30229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff9de5f90544c75afc2977c8bdc6df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32841 } }
I20260926 20:57:25.157186 30229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:25.157261 30229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff9de5f90544c75afc2977c8bdc6df3, State: Initialized, Role: FOLLOWER
I20260926 20:57:25.157864 30229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff9de5f90544c75afc2977c8bdc6df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32841 } }
I20260926 20:57:25.158070 30229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:25.158180 30229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:25.158310 30229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:25.160961 30229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff9de5f90544c75afc2977c8bdc6df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32841 } }
I20260926 20:57:25.161487 30229 leader_election.cc:302] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ff9de5f90544c75afc2977c8bdc6df3; no voters:
I20260926 20:57:25.162040 30229 leader_election.cc:288] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:25.162194 30239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:25.162914 30239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [term 1 LEADER]: Becoming Leader. State: Replica: 7ff9de5f90544c75afc2977c8bdc6df3, State: Running, Role: LEADER
I20260926 20:57:25.163479 30239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff9de5f90544c75afc2977c8bdc6df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32841 } }
I20260926 20:57:25.164214 30229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:25.166263 30241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ff9de5f90544c75afc2977c8bdc6df3. Latest consensus state: current_term: 1 leader_uuid: "7ff9de5f90544c75afc2977c8bdc6df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff9de5f90544c75afc2977c8bdc6df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32841 } } }
I20260926 20:57:25.166742 30241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:25.166384 30240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ff9de5f90544c75afc2977c8bdc6df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff9de5f90544c75afc2977c8bdc6df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32841 } } }
I20260926 20:57:25.167549 30240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:25.169636 30247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:25.174924 30247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:25.184156 30247 catalog_manager.cc:1347] Generated new cluster ID: 06ce440cdf4c4a7c91415e464716df18
I20260926 20:57:25.184307 30247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:25.196609 30247 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:25.197048 30247 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:25.211149 30247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ff9de5f90544c75afc2977c8bdc6df3: Generated new TSK 0
I20260926 20:57:25.212349 30247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:25.227283 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30162
W20260926 20:57:25.228446 30224 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52487
W20260926 20:57:25.229184 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30162: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:57:25.229329 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 30162
2026-09-26T20:57:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1889 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:25 dist-test-slave-2dg3 krb5kdc[30260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:25 dist-test-slave-2dg3 krb5kdc[30260](info): set up 2 sockets
Sep 26 20:57:25 dist-test-slave-2dg3 krb5kdc[30260](info): commencing operation
krb5kdc: starting...
W20260926 20:57:26.331259 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:26 dist-test-slave-2dg3 krb5kdc[30260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:57:26.598436 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:26.939803 30273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:26.940110 30273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:26.940285 30273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:26.947965 30273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:26.948117 30273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:26.948323 30273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:26.948424 30273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:26.948521 30273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:26.958956 30273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:26.960467 30273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:26.961941 30273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:26.974968 30281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:26.975047 30279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:26.974987 30278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:26.976439 30273 server_base.cc:1034] running on GCE node
I20260926 20:57:26.977361 30273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:26.979063 30273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:26.980307 30273 hybrid_clock.cc:648] HybridClock initialized: now 1790456246980268 us; error 39 us; skew 500 ppm
I20260926 20:57:26.980743 30273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:26 dist-test-slave-2dg3 krb5kdc[30260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:26.984217 30273 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:57:26.986413 30273 webserver.cc:460] Webserver started at http://127.0.0.1:38455/ using document root <none> and password file <none>
I20260926 20:57:26.987313 30273 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:26.987462 30273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:26.987833 30273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:26.991227 30273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "22223d512d464d0f97517bda03eaa5d5"
format_stamp: "Formatted at 2026-09-26 20:57:26 on dist-test-slave-2dg3"
server_key: "816ce38c527fdaf9febb6865cf0701bf"
server_key_iv: "6519a9093d9db605d89ebe552ba7af9e"
server_key_version: "encryptionkey@0"
I20260926 20:57:26.992120 30273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "22223d512d464d0f97517bda03eaa5d5"
format_stamp: "Formatted at 2026-09-26 20:57:26 on dist-test-slave-2dg3"
server_key: "816ce38c527fdaf9febb6865cf0701bf"
server_key_iv: "6519a9093d9db605d89ebe552ba7af9e"
server_key_version: "encryptionkey@0"
I20260926 20:57:26.998147 30273 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 20:57:27.002177 30288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:27.003754 30273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:27.003976 30273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "22223d512d464d0f97517bda03eaa5d5"
format_stamp: "Formatted at 2026-09-26 20:57:26 on dist-test-slave-2dg3"
server_key: "816ce38c527fdaf9febb6865cf0701bf"
server_key_iv: "6519a9093d9db605d89ebe552ba7af9e"
server_key_version: "encryptionkey@0"
I20260926 20:57:27.004208 30273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:27.021552 30273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:27.025849 30273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:27.026170 30273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:27.045598 30273 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36927
I20260926 20:57:27.045603 30339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36927 every 8 connection(s)
I20260926 20:57:27.047111 30273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:27.051011 30340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:27.055631 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 30273
I20260926 20:57:27.055840 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:27.056311 9190 external_mini_cluster.cc:1428] Setting key ab46c9a67855f0d3d491424fe52d2b95
I20260926 20:57:27.063767 30340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5: Bootstrap starting.
I20260926 20:57:27.067799 30340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:27.069169 30340 log.cc:824] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:27.072261 30340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5: No bootstrap required, opened a new log
Sep 26 20:57:27 dist-test-slave-2dg3 krb5kdc[30260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:57:27.083343 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:27.058308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:53373 (local address 127.0.0.1:36927)
0926 20:57:27.059157 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:27.059178 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:27.059227 (+ 49us) server_negotiation.cc:407] Connection header received
0926 20:57:27.061872 (+ 2645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:27.061903 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:27.062451 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:27.062996 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:27.064362 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:27.066406 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:27.067825 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:27.068906 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:27.069113 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:27.073751 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:27.073762 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:27.073786 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:27.073847 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:27.078221 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:27.079229 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:27.079238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:27.079247 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:27.079463 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:27.079930 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:27.079936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:27.079940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:27.080730 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:27.081092 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:27.081433 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 20:57:27.081841 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":322,"threads_started":1}
I20260926 20:57:27.083491 30340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22223d512d464d0f97517bda03eaa5d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260926 20:57:27.083948 30340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:27.084057 30340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22223d512d464d0f97517bda03eaa5d5, State: Initialized, Role: FOLLOWER
I20260926 20:57:27.084708 30340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22223d512d464d0f97517bda03eaa5d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260926 20:57:27.084992 30340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:27.085155 30340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:27.085347 30340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:27.087723 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:27.088110 30340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22223d512d464d0f97517bda03eaa5d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260926 20:57:27.088707 30340 leader_election.cc:302] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22223d512d464d0f97517bda03eaa5d5; no voters:
I20260926 20:57:27.089347 30340 leader_election.cc:288] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:27.089476 30345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:27.090655 30345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [term 1 LEADER]: Becoming Leader. State: Replica: 22223d512d464d0f97517bda03eaa5d5, State: Running, Role: LEADER
I20260926 20:57:27.091347 30345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22223d512d464d0f97517bda03eaa5d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260926 20:57:27.091831 30340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:27.093962 30347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22223d512d464d0f97517bda03eaa5d5. Latest consensus state: current_term: 1 leader_uuid: "22223d512d464d0f97517bda03eaa5d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22223d512d464d0f97517bda03eaa5d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } } }
I20260926 20:57:27.094111 30346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22223d512d464d0f97517bda03eaa5d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22223d512d464d0f97517bda03eaa5d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } } }
I20260926 20:57:27.094393 30347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:27.094391 30346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:27.099017 30355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:27.104861 30355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:27.122771 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:27.099009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:44271 (local address 127.0.0.1:36927)
0926 20:57:27.099897 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:27.099909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:27.100480 (+ 571us) server_negotiation.cc:407] Connection header received
0926 20:57:27.100693 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:27.100702 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:27.100817 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:27.100943 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:27.102979 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:27.104594 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:27.107179 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:27.108176 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:27.108350 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:27.112921 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:27.112933 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:27.112941 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:27.113026 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:27.118148 (+ 5122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:27.119420 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:27.119427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:27.119431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:27.119593 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:27.120198 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:27.120204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:27.120210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:27.120644 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:27.120914 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:27.122089 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0926 20:57:27.122277 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742}
I20260926 20:57:27.122313 30355 catalog_manager.cc:1347] Generated new cluster ID: ba9870fda6d5468e8faa7b1775d26a76
I20260926 20:57:27.123291 30355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:27.154522 30355 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:27.155957 30355 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:27.168262 30355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22223d512d464d0f97517bda03eaa5d5: Generated new TSK 0
I20260926 20:57:27.169394 30355 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:27.175424 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30273
I20260926 20:57:27.176645 30335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:27.292501 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 30273
W20260926 20:57:27.309669 30258 connection.cc:535] client connection to 127.0.0.1:36927 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36927): Connection reset by peer (error 104)
2026-09-26T20:57:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2063 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:27 dist-test-slave-2dg3 krb5kdc[30372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:27 dist-test-slave-2dg3 krb5kdc[30372](info): set up 2 sockets
Sep 26 20:57:27 dist-test-slave-2dg3 krb5kdc[30372](info): commencing operation
krb5kdc: starting...
W20260926 20:57:28.399163 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:28 dist-test-slave-2dg3 krb5kdc[30372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:57:28.664042 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:29.001982 30385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:29.002321 30385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:29.002482 30385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:29.009919 30385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:29.010051 30385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:29.010178 30385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:29.010289 30385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:29.010376 30385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:29.020560 30385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:29.021957 30385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:29.023331 30385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:29.035421 30390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:29.035449 30393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:29.035470 30391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:29.036298 30385 server_base.cc:1034] running on GCE node
I20260926 20:57:29.037071 30385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:29.038496 30385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:29.039692 30385 hybrid_clock.cc:648] HybridClock initialized: now 1790456249039622 us; error 74 us; skew 500 ppm
I20260926 20:57:29.040014 30385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:29 dist-test-slave-2dg3 krb5kdc[30372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:29.048262 30385 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:57:29.049932 30385 webserver.cc:460] Webserver started at http://127.0.0.1:46581/ using document root <none> and password file <none>
I20260926 20:57:29.050670 30385 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:29.050778 30385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:29.051137 30385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:29.054224 30385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "5fe0291a3c9d4f14af727f85b5c68a18"
format_stamp: "Formatted at 2026-09-26 20:57:29 on dist-test-slave-2dg3"
server_key: "250e06cb9606f93c826c72fee3c1815b"
server_key_iv: "4752c1442f928e9f3084765ea46331be"
server_key_version: "encryptionkey@0"
I20260926 20:57:29.055060 30385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "5fe0291a3c9d4f14af727f85b5c68a18"
format_stamp: "Formatted at 2026-09-26 20:57:29 on dist-test-slave-2dg3"
server_key: "250e06cb9606f93c826c72fee3c1815b"
server_key_iv: "4752c1442f928e9f3084765ea46331be"
server_key_version: "encryptionkey@0"
I20260926 20:57:29.060791 30385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:57:29.064702 30400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:29.066282 30385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:57:29.066458 30385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "5fe0291a3c9d4f14af727f85b5c68a18"
format_stamp: "Formatted at 2026-09-26 20:57:29 on dist-test-slave-2dg3"
server_key: "250e06cb9606f93c826c72fee3c1815b"
server_key_iv: "4752c1442f928e9f3084765ea46331be"
server_key_version: "encryptionkey@0"
I20260926 20:57:29.066663 30385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:29.080590 30385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:29.084894 30385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:29.085134 30385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:29.104065 30385 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44567
I20260926 20:57:29.104099 30451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44567 every 8 connection(s)
I20260926 20:57:29.105543 30385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:29.109524 30452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:29.110926 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 30385
I20260926 20:57:29.111138 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:29.111675 9190 external_mini_cluster.cc:1428] Setting key 0f242ce1bc2cd316a84658d4c9ebab71
I20260926 20:57:29.122592 30452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18: Bootstrap starting.
I20260926 20:57:29.125030 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:29.113580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:45775 (local address 127.0.0.1:44567)
0926 20:57:29.114343 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:29.114355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:29.114398 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:57:29.116206 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:29.116232 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:29.116674 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:29.117081 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:29.117636 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:29.118963 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:29.120316 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:29.121017 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:29.121225 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:29.122582 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:29.122591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:29.122614 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:29.123300 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:29.123389 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:29.123417 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:29.123594 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:29.123860 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 20:57:29.124032 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":323,"threads_started":1}
I20260926 20:57:29.126062 30452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:29.127218 30452 log.cc:824] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:29.129267 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:29.130617 30452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18: No bootstrap required, opened a new log
I20260926 20:57:29.141033 30452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe0291a3c9d4f14af727f85b5c68a18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } }
I20260926 20:57:29.141562 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:29.134690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:42437 (local address 127.0.0.1:44567)
0926 20:57:29.134932 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:29.134942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:29.135265 (+ 323us) server_negotiation.cc:407] Connection header received
0926 20:57:29.135433 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:29.135439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:29.135519 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:29.135628 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:29.136115 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:29.137302 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:29.138769 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:29.139534 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:29.139673 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:29.140807 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:29.140813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:29.140817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:29.140838 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:29.140904 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:29.140923 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:29.141048 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:29.141204 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 20:57:29.141259 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:57:29.141585 30452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:29.141724 30452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe0291a3c9d4f14af727f85b5c68a18, State: Initialized, Role: FOLLOWER
I20260926 20:57:29.142432 30452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe0291a3c9d4f14af727f85b5c68a18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } }
I20260926 20:57:29.142692 30452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:29.142915 30452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:29.143098 30452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:29.145310 30452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe0291a3c9d4f14af727f85b5c68a18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } }
I20260926 20:57:29.145860 30452 leader_election.cc:302] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fe0291a3c9d4f14af727f85b5c68a18; no voters:
I20260926 20:57:29.146507 30452 leader_election.cc:288] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:29.146716 30462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:29.147748 30462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [term 1 LEADER]: Becoming Leader. State: Replica: 5fe0291a3c9d4f14af727f85b5c68a18, State: Running, Role: LEADER
I20260926 20:57:29.148360 30462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe0291a3c9d4f14af727f85b5c68a18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } }
I20260926 20:57:29.148887 30452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:29.151821 30464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fe0291a3c9d4f14af727f85b5c68a18. Latest consensus state: current_term: 1 leader_uuid: "5fe0291a3c9d4f14af727f85b5c68a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe0291a3c9d4f14af727f85b5c68a18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } } }
I20260926 20:57:29.152344 30464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:29.152881 30463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fe0291a3c9d4f14af727f85b5c68a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe0291a3c9d4f14af727f85b5c68a18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } } }
I20260926 20:57:29.153414 30463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:29.155860 30469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:29.161885 30469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:29.172518 30469 catalog_manager.cc:1347] Generated new cluster ID: 0346dcf9a9ce4287b572ce10a8c80c8c
I20260926 20:57:29.172746 30469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:29.180651 30469 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:29.182230 30469 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:29.188172 30469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fe0291a3c9d4f14af727f85b5c68a18: Generated new TSK 0
I20260926 20:57:29.189034 30469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:29.205648 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30385
W20260926 20:57:29.206996 30447 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.82:45775
W20260926 20:57:29.207890 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30385: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:57:29.208098 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 30385
W20260926 20:57:29.227101 30370 connection.cc:535] client connection to 127.0.0.1:44567 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44567): Connection reset by peer (error 104)
2026-09-26T20:57:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:29 dist-test-slave-2dg3 krb5kdc[30483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:29 dist-test-slave-2dg3 krb5kdc[30483](info): set up 2 sockets
Sep 26 20:57:29 dist-test-slave-2dg3 krb5kdc[30483](info): commencing operation
krb5kdc: starting...
W20260926 20:57:30.319303 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:30 dist-test-slave-2dg3 krb5kdc[30483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:57:30.603193 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:30.935894 30496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:30.936197 30496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:30.936292 30496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:30.943607 30496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:30.943724 30496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:30.943835 30496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:30.943938 30496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:30.944017 30496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:30.954169 30496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:30.955521 30496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:30.956867 30496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:30.968724 30501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:30.968755 30502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:30.968681 30504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:30.969427 30496 server_base.cc:1034] running on GCE node
I20260926 20:57:30.970209 30496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:30.971624 30496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:30.972831 30496 hybrid_clock.cc:648] HybridClock initialized: now 1790456250972799 us; error 31 us; skew 500 ppm
I20260926 20:57:30.973186 30496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:30 dist-test-slave-2dg3 krb5kdc[30483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:30.976703 30496 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:57:30.978116 30496 webserver.cc:460] Webserver started at http://127.0.0.1:41823/ using document root <none> and password file <none>
I20260926 20:57:30.978847 30496 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:30.979003 30496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:30.979298 30496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:30.982280 30496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "e10cab96f1d04979a55630e1337a312a"
format_stamp: "Formatted at 2026-09-26 20:57:30 on dist-test-slave-2dg3"
server_key: "94a1c9992ce0a31af3635585f86d4502"
server_key_iv: "a9eb74b63a95235647f53626dc378b57"
server_key_version: "encryptionkey@0"
I20260926 20:57:30.983165 30496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "e10cab96f1d04979a55630e1337a312a"
format_stamp: "Formatted at 2026-09-26 20:57:30 on dist-test-slave-2dg3"
server_key: "94a1c9992ce0a31af3635585f86d4502"
server_key_iv: "a9eb74b63a95235647f53626dc378b57"
server_key_version: "encryptionkey@0"
I20260926 20:57:30.989972 30496 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260926 20:57:30.994287 30511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:30.995729 30496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:30.995918 30496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "e10cab96f1d04979a55630e1337a312a"
format_stamp: "Formatted at 2026-09-26 20:57:30 on dist-test-slave-2dg3"
server_key: "94a1c9992ce0a31af3635585f86d4502"
server_key_iv: "a9eb74b63a95235647f53626dc378b57"
server_key_version: "encryptionkey@0"
I20260926 20:57:30.996131 30496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:31.001680 30496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:31.005565 30496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:31.005867 30496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:31.024441 30496 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34569
I20260926 20:57:31.024479 30562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34569 every 8 connection(s)
I20260926 20:57:31.025894 30496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:31.029412 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 30496
I20260926 20:57:31.029614 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:31.030200 9190 external_mini_cluster.cc:1428] Setting key be8be3b306ca8930d9497fafd2476f28
I20260926 20:57:31.030236 30563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:31.038601 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:31.032170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:33219 (local address 127.0.0.1:34569)
0926 20:57:31.032992 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:31.033004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:31.033049 (+ 45us) server_negotiation.cc:407] Connection header received
0926 20:57:31.035134 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:31.035158 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:31.035519 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:31.035859 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:31.037031 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:31.037037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:31.037049 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:31.037067 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:31.037131 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:31.037151 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:31.037326 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:31.037431 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 20:57:31.037648 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":359,"threads_started":1}
I20260926 20:57:31.042260 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:31.044230 30563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a: Bootstrap starting.
I20260926 20:57:31.048277 30563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:31.049477 30563 log.cc:824] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:31.050279 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:31.047571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:59225 (local address 127.0.0.1:34569)
0926 20:57:31.047755 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:31.047760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:31.048248 (+ 488us) server_negotiation.cc:407] Connection header received
0926 20:57:31.048392 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:31.048398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:31.048466 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:31.048574 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:31.049526 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:31.049531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:31.049536 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:31.049553 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:31.049609 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:31.049630 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:31.049746 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:31.049836 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 20:57:31.049897 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:57:31.053114 30563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a: No bootstrap required, opened a new log
I20260926 20:57:31.060725 30563 raft_consensus.cc:348] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e10cab96f1d04979a55630e1337a312a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34569 } }
I20260926 20:57:31.061136 30563 raft_consensus.cc:374] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:31.061251 30563 raft_consensus.cc:729] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e10cab96f1d04979a55630e1337a312a, State: Initialized, Role: FOLLOWER
I20260926 20:57:31.061883 30563 consensus_queue.cc:260] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e10cab96f1d04979a55630e1337a312a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34569 } }
I20260926 20:57:31.062124 30563 raft_consensus.cc:388] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:31.062274 30563 raft_consensus.cc:482] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:31.062441 30563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:31.064973 30563 raft_consensus.cc:504] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e10cab96f1d04979a55630e1337a312a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34569 } }
I20260926 20:57:31.065543 30563 leader_election.cc:302] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e10cab96f1d04979a55630e1337a312a; no voters:
I20260926 20:57:31.066071 30563 leader_election.cc:288] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:31.066234 30573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:31.066920 30573 raft_consensus.cc:686] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [term 1 LEADER]: Becoming Leader. State: Replica: e10cab96f1d04979a55630e1337a312a, State: Running, Role: LEADER
I20260926 20:57:31.067435 30573 consensus_queue.cc:237] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e10cab96f1d04979a55630e1337a312a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34569 } }
I20260926 20:57:31.068181 30563 sys_catalog.cc:564] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:31.070283 30575 sys_catalog.cc:455] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e10cab96f1d04979a55630e1337a312a. Latest consensus state: current_term: 1 leader_uuid: "e10cab96f1d04979a55630e1337a312a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e10cab96f1d04979a55630e1337a312a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34569 } } }
I20260926 20:57:31.070757 30575 sys_catalog.cc:458] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:31.071131 30574 sys_catalog.cc:455] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e10cab96f1d04979a55630e1337a312a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e10cab96f1d04979a55630e1337a312a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34569 } } }
I20260926 20:57:31.071547 30574 sys_catalog.cc:458] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:31.075389 30580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:31.080682 30580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:31.090217 30580 catalog_manager.cc:1347] Generated new cluster ID: a43001960e5d4362b74eed0bc932c04d
I20260926 20:57:31.090363 30580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:31.101227 30580 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:31.101681 30580 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:31.107645 30580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e10cab96f1d04979a55630e1337a312a: Generated new TSK 0
I20260926 20:57:31.108470 30580 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:31.121191 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30496
W20260926 20:57:31.123046 30558 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.82:33219
W20260926 20:57:31.123804 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30496: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:57:31.123975 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 30496
2026-09-26T20:57:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:31 dist-test-slave-2dg3 krb5kdc[30594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:31 dist-test-slave-2dg3 krb5kdc[30594](info): set up 2 sockets
Sep 26 20:57:31 dist-test-slave-2dg3 krb5kdc[30594](info): commencing operation
krb5kdc: starting...
W20260926 20:57:32.231463 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:32 dist-test-slave-2dg3 krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:57:32.502377 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:32.849313 30607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:32.849635 30607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:32.849841 30607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:32.857219 30607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:32.857383 30607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:32.857507 30607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:32.857621 30607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:32.857714 30607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:32.868033 30607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:32.869508 30607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:32.870967 30607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:32.883771 30615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:32.883846 30612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:32.883751 30613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:32.884644 30607 server_base.cc:1034] running on GCE node
I20260926 20:57:32.885385 30607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:32.886770 30607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:32.888036 30607 hybrid_clock.cc:648] HybridClock initialized: now 1790456252887974 us; error 50 us; skew 500 ppm
I20260926 20:57:32.888386 30607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:32 dist-test-slave-2dg3 krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:32.891587 30607 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:57:32.893098 30607 webserver.cc:460] Webserver started at http://127.0.0.1:42453/ using document root <none> and password file <none>
I20260926 20:57:32.893808 30607 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:32.893951 30607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:32.894243 30607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:32.897378 30607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "d0c0bb0ffd67430e8531f3a907e2b64e"
format_stamp: "Formatted at 2026-09-26 20:57:32 on dist-test-slave-2dg3"
server_key: "63ca7e1c105d7640c76780cafbddabea"
server_key_iv: "577f046d197e11ac578aeecc06355efb"
server_key_version: "encryptionkey@0"
I20260926 20:57:32.898178 30607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "d0c0bb0ffd67430e8531f3a907e2b64e"
format_stamp: "Formatted at 2026-09-26 20:57:32 on dist-test-slave-2dg3"
server_key: "63ca7e1c105d7640c76780cafbddabea"
server_key_iv: "577f046d197e11ac578aeecc06355efb"
server_key_version: "encryptionkey@0"
I20260926 20:57:32.903460 30607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 20:57:32.907111 30622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:32.908401 30607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:57:32.908601 30607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "d0c0bb0ffd67430e8531f3a907e2b64e"
format_stamp: "Formatted at 2026-09-26 20:57:32 on dist-test-slave-2dg3"
server_key: "63ca7e1c105d7640c76780cafbddabea"
server_key_iv: "577f046d197e11ac578aeecc06355efb"
server_key_version: "encryptionkey@0"
I20260926 20:57:32.908803 30607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:32.925873 30607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:32.930907 30607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:32.931257 30607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:32.952435 30607 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40281
I20260926 20:57:32.952476 30673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40281 every 8 connection(s)
I20260926 20:57:32.953847 30607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:32.957976 30674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:32.958659 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 30607
I20260926 20:57:32.958880 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:32.959457 9190 external_mini_cluster.cc:1428] Setting key 49e054363a775c6aed4daae0d1f781c0
I20260926 20:57:32.971271 30674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e: Bootstrap starting.
Sep 26 20:57:32 dist-test-slave-2dg3 krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:57:32.974701 30674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:32.975806 30674 log.cc:824] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:32.979373 30674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e: No bootstrap required, opened a new log
I20260926 20:57:32.984815 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:32.961447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:49343 (local address 127.0.0.1:40281)
0926 20:57:32.962241 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:32.962254 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:32.962294 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:57:32.964875 (+ 2581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:32.964904 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:32.965358 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:32.965779 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:32.967297 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:32.968638 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:32.969937 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:32.970628 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:32.970864 (+ 236us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:32.975405 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:32.975415 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:32.975431 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:32.975469 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:32.979394 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:32.980420 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:32.980430 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:32.980438 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:32.980667 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:32.981221 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:32.981227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:32.981230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:32.982004 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:32.982379 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:32.982876 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 20:57:32.983267 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":339,"threads_started":1}
I20260926 20:57:32.989266 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:32.989883 30674 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c0bb0ffd67430e8531f3a907e2b64e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260926 20:57:32.990291 30674 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:32.990398 30674 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0c0bb0ffd67430e8531f3a907e2b64e, State: Initialized, Role: FOLLOWER
I20260926 20:57:32.991068 30674 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c0bb0ffd67430e8531f3a907e2b64e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260926 20:57:32.991325 30674 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:32.991467 30674 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:32.991640 30674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:32.993867 30674 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c0bb0ffd67430e8531f3a907e2b64e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260926 20:57:32.994606 30674 leader_election.cc:302] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0c0bb0ffd67430e8531f3a907e2b64e; no voters:
I20260926 20:57:32.995321 30674 leader_election.cc:288] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:32.995510 30684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:32.997015 30684 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [term 1 LEADER]: Becoming Leader. State: Replica: d0c0bb0ffd67430e8531f3a907e2b64e, State: Running, Role: LEADER
I20260926 20:57:32.997823 30674 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:32.997761 30684 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c0bb0ffd67430e8531f3a907e2b64e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260926 20:57:33.008833 30686 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0c0bb0ffd67430e8531f3a907e2b64e. Latest consensus state: current_term: 1 leader_uuid: "d0c0bb0ffd67430e8531f3a907e2b64e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c0bb0ffd67430e8531f3a907e2b64e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } } }
I20260926 20:57:33.009811 30686 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:33.009586 30685 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0c0bb0ffd67430e8531f3a907e2b64e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c0bb0ffd67430e8531f3a907e2b64e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } } }
I20260926 20:57:33.009948 30685 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:33.012192 30695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:33.016842 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:32.994178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:41539 (local address 127.0.0.1:40281)
0926 20:57:32.994387 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:32.994397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:32.994897 (+ 500us) server_negotiation.cc:407] Connection header received
0926 20:57:32.995154 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:32.995162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:32.995265 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:32.995392 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:32.997393 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:32.998618 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.001007 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:33.001939 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:33.002138 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:33.006515 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:33.006525 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:33.006533 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:33.006591 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:33.013221 (+ 6630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:33.014341 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:33.014346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:33.014350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:33.014507 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:33.015048 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:33.015053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:33.015057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:33.015499 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:33.015800 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:33.016271 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 20:57:33.016468 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:57:33.017762 30695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:33.028216 30695 catalog_manager.cc:1347] Generated new cluster ID: 230adf88a9a14fd293044bb62cc286ac
I20260926 20:57:33.028374 30695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:33.037276 30695 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:33.039141 30695 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:33.060774 30695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0c0bb0ffd67430e8531f3a907e2b64e: Generated new TSK 0
I20260926 20:57:33.061914 30695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:33.076378 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30607
I20260926 20:57:33.077836 30669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:33.172067 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 30607
W20260926 20:57:33.189643 30592 connection.cc:535] client connection to 127.0.0.1:40281 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40281): Connection reset by peer (error 104)
2026-09-26T20:57:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2053 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:33 dist-test-slave-2dg3 krb5kdc[30706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:33 dist-test-slave-2dg3 krb5kdc[30706](info): set up 2 sockets
Sep 26 20:57:33 dist-test-slave-2dg3 krb5kdc[30706](info): commencing operation
krb5kdc: starting...
W20260926 20:57:34.275375 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:34 dist-test-slave-2dg3 krb5kdc[30706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:57:34.536369 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:34.868530 30719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:34.868860 30719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:34.869045 30719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:34.876510 30719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:34.876691 30719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:34.876807 30719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:34.876931 30719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:34.877032 30719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:34.887852 30719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46707
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:34.889278 30719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:34.890650 30719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:34.902454 30724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:34.902498 30727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:34.902535 30725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:34.903900 30719 server_base.cc:1034] running on GCE node
I20260926 20:57:34.904606 30719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:34.905925 30719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:34.907152 30719 hybrid_clock.cc:648] HybridClock initialized: now 1790456254907100 us; error 38 us; skew 500 ppm
I20260926 20:57:34.907536 30719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:34 dist-test-slave-2dg3 krb5kdc[30706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:34.915719 30719 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:57:34.917394 30719 webserver.cc:460] Webserver started at http://127.0.0.1:42365/ using document root <none> and password file <none>
I20260926 20:57:34.918098 30719 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:34.918249 30719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:34.918581 30719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:34.921723 30719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "4d32dc3f757c439d9c76b02729539348"
format_stamp: "Formatted at 2026-09-26 20:57:34 on dist-test-slave-2dg3"
server_key: "c9ba38bc6e14ab41c7eea60f3ea76c74"
server_key_iv: "08672f8ddebc70a47279af0e1dbad56d"
server_key_version: "encryptionkey@0"
I20260926 20:57:34.922540 30719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "4d32dc3f757c439d9c76b02729539348"
format_stamp: "Formatted at 2026-09-26 20:57:34 on dist-test-slave-2dg3"
server_key: "c9ba38bc6e14ab41c7eea60f3ea76c74"
server_key_iv: "08672f8ddebc70a47279af0e1dbad56d"
server_key_version: "encryptionkey@0"
I20260926 20:57:34.928211 30719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260926 20:57:34.931790 30734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:34.933108 30719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:57:34.933293 30719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "4d32dc3f757c439d9c76b02729539348"
format_stamp: "Formatted at 2026-09-26 20:57:34 on dist-test-slave-2dg3"
server_key: "c9ba38bc6e14ab41c7eea60f3ea76c74"
server_key_iv: "08672f8ddebc70a47279af0e1dbad56d"
server_key_version: "encryptionkey@0"
I20260926 20:57:34.933496 30719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:34.956223 30719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:34.960731 30719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:34.961066 30719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:34.981261 30719 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46707
I20260926 20:57:34.981278 30785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46707 every 8 connection(s)
I20260926 20:57:34.982834 30719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:34.986598 30786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:34.992578 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 30719
I20260926 20:57:34.992763 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:34.993243 9190 external_mini_cluster.cc:1428] Setting key e3901296443e816bedc48c25148d465e
I20260926 20:57:34.997555 30786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348: Bootstrap starting.
I20260926 20:57:35.000950 30786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:35.002223 30786 log.cc:824] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:35.005697 30786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348: No bootstrap required, opened a new log
I20260926 20:57:35.007793 30789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:34.995284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:56727 (local address 127.0.0.1:46707)
0926 20:57:34.996104 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:34.996115 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:34.996155 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:57:34.997884 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:34.997906 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:34.998297 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:34.998608 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:34.999310 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:35.000677 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:35.002548 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:35.003422 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:35.003616 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:35.004817 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:35.004829 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:35.004848 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:35.005612 (+ 764us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:35.005725 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:35.005763 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:35.005945 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:35.006190 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 20:57:35.006395 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":395,"threads_started":1}
I20260926 20:57:35.012023 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:35.017189 30786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d32dc3f757c439d9c76b02729539348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46707 } }
I20260926 20:57:35.017849 30786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:35.017988 30786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d32dc3f757c439d9c76b02729539348, State: Initialized, Role: FOLLOWER
I20260926 20:57:35.018748 30786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d32dc3f757c439d9c76b02729539348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46707 } }
I20260926 20:57:35.019102 30786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:35.019284 30786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:35.019474 30786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:35.022470 30786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d32dc3f757c439d9c76b02729539348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46707 } }
I20260926 20:57:35.023146 30786 leader_election.cc:302] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d32dc3f757c439d9c76b02729539348; no voters:
I20260926 20:57:35.024035 30786 leader_election.cc:288] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:35.024221 30796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:35.025345 30789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:35.017719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:51133 (local address 127.0.0.1:46707)
0926 20:57:35.017969 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:35.017983 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:35.018549 (+ 566us) server_negotiation.cc:407] Connection header received
0926 20:57:35.018696 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:35.018703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:35.018829 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:35.018966 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:35.019586 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:35.020790 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:35.022090 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:35.022923 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:35.023070 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:35.024219 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:35.024226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:35.024233 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:35.024257 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:35.024363 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:35.024394 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:35.024565 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:35.024762 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 20:57:35.024840 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 20:57:35.027015 30786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:35.031302 30796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [term 1 LEADER]: Becoming Leader. State: Replica: 4d32dc3f757c439d9c76b02729539348, State: Running, Role: LEADER
I20260926 20:57:35.031840 30796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d32dc3f757c439d9c76b02729539348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46707 } }
I20260926 20:57:35.034519 30797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d32dc3f757c439d9c76b02729539348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d32dc3f757c439d9c76b02729539348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46707 } } }
I20260926 20:57:35.034704 30798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d32dc3f757c439d9c76b02729539348. Latest consensus state: current_term: 1 leader_uuid: "4d32dc3f757c439d9c76b02729539348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d32dc3f757c439d9c76b02729539348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46707 } } }
I20260926 20:57:35.034961 30798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:35.034961 30797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348 [sys.catalog]: This master's current role is: LEADER
W20260926 20:57:35.040126 30810 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:57:35.040261 30810 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:57:35.040401 30811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:35.042192 30811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:35.051088 30811 catalog_manager.cc:1347] Generated new cluster ID: b9916fee857a4ad78008c4c5c26ccc32
I20260926 20:57:35.051227 30811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:35.082077 30811 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:35.083534 30811 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:35.094833 30811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d32dc3f757c439d9c76b02729539348: Generated new TSK 0
I20260926 20:57:35.095708 30811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:35.106822 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30719
W20260926 20:57:35.107937 30781 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.82:56727
W20260926 20:57:35.108760 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30719: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:57:35.108937 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 30719
W20260926 20:57:35.125859 30704 connection.cc:535] client connection to 127.0.0.1:46707 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46707): Connection reset by peer (error 104)
2026-09-26T20:57:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:35 dist-test-slave-2dg3 krb5kdc[30817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:35 dist-test-slave-2dg3 krb5kdc[30817](info): set up 2 sockets
Sep 26 20:57:35 dist-test-slave-2dg3 krb5kdc[30817](info): commencing operation
krb5kdc: starting...
W20260926 20:57:36.203438 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:36 dist-test-slave-2dg3 krb5kdc[30817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:57:36.470382 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:36.806788 30830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:36.807154 30830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:36.807246 30830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:36.814436 30830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:36.814570 30830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:36.814620 30830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:36.814673 30830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:36.814738 30830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:36.824730 30830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38925
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:36.826113 30830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:36.827445 30830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:36.839270 30838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:36.839454 30836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:36.839270 30835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:36.840281 30830 server_base.cc:1034] running on GCE node
I20260926 20:57:36.841104 30830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:36.842406 30830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:36.843592 30830 hybrid_clock.cc:648] HybridClock initialized: now 1790456256843526 us; error 56 us; skew 500 ppm
I20260926 20:57:36.843895 30830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:36 dist-test-slave-2dg3 krb5kdc[30817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:36.847074 30830 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:57:36.848429 30830 webserver.cc:460] Webserver started at http://127.0.0.1:38421/ using document root <none> and password file <none>
I20260926 20:57:36.849088 30830 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:36.849236 30830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:36.849545 30830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:36.852537 30830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "abdab306805a4d8eb3b8db1674f26f4c"
format_stamp: "Formatted at 2026-09-26 20:57:36 on dist-test-slave-2dg3"
server_key: "2c9955f1e80d58e61f32ce91f636ab34"
server_key_iv: "dd62292d2ee7e5bd747e687eff3f5223"
server_key_version: "encryptionkey@0"
I20260926 20:57:36.853296 30830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "abdab306805a4d8eb3b8db1674f26f4c"
format_stamp: "Formatted at 2026-09-26 20:57:36 on dist-test-slave-2dg3"
server_key: "2c9955f1e80d58e61f32ce91f636ab34"
server_key_iv: "dd62292d2ee7e5bd747e687eff3f5223"
server_key_version: "encryptionkey@0"
I20260926 20:57:36.858558 30830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:57:36.862134 30845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:36.863731 30830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:36.863910 30830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "abdab306805a4d8eb3b8db1674f26f4c"
format_stamp: "Formatted at 2026-09-26 20:57:36 on dist-test-slave-2dg3"
server_key: "2c9955f1e80d58e61f32ce91f636ab34"
server_key_iv: "dd62292d2ee7e5bd747e687eff3f5223"
server_key_version: "encryptionkey@0"
I20260926 20:57:36.864128 30830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:36.869942 30830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:36.874270 30830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:36.874584 30830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:36.893553 30830 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38925
I20260926 20:57:36.893591 30896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38925 every 8 connection(s)
I20260926 20:57:36.895081 30830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:36.896490 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 30830
I20260926 20:57:36.896703 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:36.897135 9190 external_mini_cluster.cc:1428] Setting key 06b37fdbc22772cc3518e4bbdc1c811e
I20260926 20:57:36.899528 30897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:36.906013 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:36.899005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:33021 (local address 127.0.0.1:38925)
0926 20:57:36.899775 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:36.899791 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:36.899845 (+ 54us) server_negotiation.cc:407] Connection header received
0926 20:57:36.902152 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:36.902178 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:36.902645 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:36.903098 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:36.904073 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:36.904082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:36.904098 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:36.904115 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:36.904207 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:36.904244 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:36.904474 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:36.904611 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 20:57:36.904802 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":293,"threads_started":1}
I20260926 20:57:36.909626 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:36.912647 30897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c: Bootstrap starting.
I20260926 20:57:36.916013 30897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:36.916680 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:36.913978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:43527 (local address 127.0.0.1:38925)
0926 20:57:36.914194 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:36.914206 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:36.914661 (+ 455us) server_negotiation.cc:407] Connection header received
0926 20:57:36.914785 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:36.914803 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:36.914884 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:36.914982 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:36.915907 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:36.915913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:36.915919 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:36.915938 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:36.915986 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:36.916013 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:36.916136 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:36.916292 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 20:57:36.916350 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:57:36.917268 30897 log.cc:824] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:36.920051 30897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c: No bootstrap required, opened a new log
I20260926 20:57:36.927023 30897 raft_consensus.cc:348] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdab306805a4d8eb3b8db1674f26f4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260926 20:57:36.927386 30897 raft_consensus.cc:374] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:36.927489 30897 raft_consensus.cc:729] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abdab306805a4d8eb3b8db1674f26f4c, State: Initialized, Role: FOLLOWER
I20260926 20:57:36.928010 30897 consensus_queue.cc:260] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdab306805a4d8eb3b8db1674f26f4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260926 20:57:36.928234 30897 raft_consensus.cc:388] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:36.928370 30897 raft_consensus.cc:482] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:36.928534 30897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:36.930622 30897 raft_consensus.cc:504] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdab306805a4d8eb3b8db1674f26f4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260926 20:57:36.931125 30897 leader_election.cc:302] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abdab306805a4d8eb3b8db1674f26f4c; no voters:
I20260926 20:57:36.931694 30897 leader_election.cc:288] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:36.931823 30907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:36.932932 30907 raft_consensus.cc:686] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [term 1 LEADER]: Becoming Leader. State: Replica: abdab306805a4d8eb3b8db1674f26f4c, State: Running, Role: LEADER
I20260926 20:57:36.933529 30907 consensus_queue.cc:237] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdab306805a4d8eb3b8db1674f26f4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } }
I20260926 20:57:36.933862 30897 sys_catalog.cc:564] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:36.936523 30908 sys_catalog.cc:455] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abdab306805a4d8eb3b8db1674f26f4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdab306805a4d8eb3b8db1674f26f4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } } }
I20260926 20:57:36.936977 30908 sys_catalog.cc:458] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:36.937211 30909 sys_catalog.cc:455] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader abdab306805a4d8eb3b8db1674f26f4c. Latest consensus state: current_term: 1 leader_uuid: "abdab306805a4d8eb3b8db1674f26f4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdab306805a4d8eb3b8db1674f26f4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38925 } } }
I20260926 20:57:36.937531 30909 sys_catalog.cc:458] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:36.940042 30916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:36.945240 30916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:36.954348 30916 catalog_manager.cc:1347] Generated new cluster ID: 574bd8d571fe4e259681c0306ee3b114
I20260926 20:57:36.954480 30916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:36.979094 30916 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:36.979539 30916 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:36.995699 30916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abdab306805a4d8eb3b8db1674f26f4c: Generated new TSK 0
I20260926 20:57:36.996591 30916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:37.004078 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30830
W20260926 20:57:37.005146 30892 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.82:33021
W20260926 20:57:37.005877 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30830: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:57:37.006050 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 30830
2026-09-26T20:57:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1896 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:37 dist-test-slave-2dg3 krb5kdc[30928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:37 dist-test-slave-2dg3 krb5kdc[30928](info): set up 2 sockets
Sep 26 20:57:37 dist-test-slave-2dg3 krb5kdc[30928](info): commencing operation
krb5kdc: starting...
W20260926 20:57:38.107336 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:38 dist-test-slave-2dg3 krb5kdc[30928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:57:38.368381 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:46379
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:39897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:46379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:38.699035 30941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:38.699445 30941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:38.699656 30941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:38.708314 30941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:38.708441 30941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:38.708488 30941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:38.708585 30941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:38.708650 30941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:38.718777 30941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:46379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:46379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:38.720188 30941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:38.721575 30941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:38.733820 30946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:38.733814 30947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:38.733873 30949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:38.734640 30941 server_base.cc:1034] running on GCE node
I20260926 20:57:38.735543 30941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:38.737030 30941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:38.738299 30941 hybrid_clock.cc:648] HybridClock initialized: now 1790456258738237 us; error 41 us; skew 500 ppm
I20260926 20:57:38.738713 30941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:38 dist-test-slave-2dg3 krb5kdc[30928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456258, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:38.742194 30941 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:57:38.743779 30941 webserver.cc:460] Webserver started at http://127.8.249.190:43885/ using document root <none> and password file <none>
I20260926 20:57:38.744442 30941 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:38.744554 30941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:38.744817 30941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:38.747959 30941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "a2ba9c672bbe49828f7817417ebe7b4e"
format_stamp: "Formatted at 2026-09-26 20:57:38 on dist-test-slave-2dg3"
server_key: "84b03c6e07add8446e6cf75b525dfb4d"
server_key_iv: "21d517921eb9c73260f4c43a4251470d"
server_key_version: "encryptionkey@0"
I20260926 20:57:38.748745 30941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "a2ba9c672bbe49828f7817417ebe7b4e"
format_stamp: "Formatted at 2026-09-26 20:57:38 on dist-test-slave-2dg3"
server_key: "84b03c6e07add8446e6cf75b525dfb4d"
server_key_iv: "21d517921eb9c73260f4c43a4251470d"
server_key_version: "encryptionkey@0"
I20260926 20:57:38.754041 30941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:57:38.757669 30956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:38.759101 30941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:38.759251 30941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "a2ba9c672bbe49828f7817417ebe7b4e"
format_stamp: "Formatted at 2026-09-26 20:57:38 on dist-test-slave-2dg3"
server_key: "84b03c6e07add8446e6cf75b525dfb4d"
server_key_iv: "21d517921eb9c73260f4c43a4251470d"
server_key_version: "encryptionkey@0"
I20260926 20:57:38.759426 30941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:38.780135 30941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:38.784937 30941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:38.785183 30941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:38.804275 30941 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:46379
I20260926 20:57:38.804342 31007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:46379 every 8 connection(s)
I20260926 20:57:38.805644 30941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:38.809429 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:38.814729 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 30941
I20260926 20:57:38.814932 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:38.815357 9190 external_mini_cluster.cc:1428] Setting key ae9a16442d87f26e4446dd717877d167
I20260926 20:57:38.820168 31008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e: Bootstrap starting.
I20260926 20:57:38.823989 31008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:38.825270 31008 log.cc:824] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:38.828480 31008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e: No bootstrap required, opened a new log
Sep 26 20:57:38 dist-test-slave-2dg3 krb5kdc[30928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:38.838920 31008 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ba9c672bbe49828f7817417ebe7b4e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46379 } }
I20260926 20:57:38.839387 31008 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:38.839509 31008 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ba9c672bbe49828f7817417ebe7b4e, State: Initialized, Role: FOLLOWER
I20260926 20:57:38.839947 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:38.817251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53401 (local address 127.8.249.190:46379)
0926 20:57:38.818129 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:38.818143 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:38.818190 (+ 47us) server_negotiation.cc:407] Connection header received
0926 20:57:38.819875 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:38.819897 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:38.820337 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:38.820654 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:38.822467 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:38.823823 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:38.825612 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:38.826321 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:38.826456 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:38.830724 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:38.830736 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:38.830756 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:38.830834 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:38.834910 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:38.835778 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:38.835787 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:38.835797 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:38.836015 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:38.836567 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:38.836572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:38.836575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:38.837348 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:38.837674 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:38.838127 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 20:57:38.838465 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":418,"threads_started":1}
I20260926 20:57:38.840196 31008 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ba9c672bbe49828f7817417ebe7b4e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46379 } }
I20260926 20:57:38.840517 31008 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:38.840679 31008 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:38.840857 31008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:38.843575 31008 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ba9c672bbe49828f7817417ebe7b4e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46379 } }
I20260926 20:57:38.844105 31008 leader_election.cc:302] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2ba9c672bbe49828f7817417ebe7b4e; no voters:
I20260926 20:57:38.844324 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:38.844735 31008 leader_election.cc:288] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:38.844904 31013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:38.847100 31008 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:38.851992 31013 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [term 1 LEADER]: Becoming Leader. State: Replica: a2ba9c672bbe49828f7817417ebe7b4e, State: Running, Role: LEADER
I20260926 20:57:38.852708 31013 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ba9c672bbe49828f7817417ebe7b4e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46379 } }
I20260926 20:57:38.856439 31014 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2ba9c672bbe49828f7817417ebe7b4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ba9c672bbe49828f7817417ebe7b4e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46379 } } }
I20260926 20:57:38.859040 31015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2ba9c672bbe49828f7817417ebe7b4e. Latest consensus state: current_term: 1 leader_uuid: "a2ba9c672bbe49828f7817417ebe7b4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ba9c672bbe49828f7817417ebe7b4e" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46379 } } }
I20260926 20:57:38.859616 31014 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:38.859642 31015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e [sys.catalog]: This master's current role is: LEADER
W20260926 20:57:38.866885 31032 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:57:38.867043 31032 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:57:38.867210 31033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:38.869005 31033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:38.874150 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:38.853620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49715 (local address 127.8.249.190:46379)
0926 20:57:38.853863 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:38.853873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:38.854216 (+ 343us) server_negotiation.cc:407] Connection header received
0926 20:57:38.854479 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:38.854486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:38.854576 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:38.854684 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:38.856621 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:38.861059 (+ 4438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:38.862775 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:38.863243 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:38.863340 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:38.866488 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:38.866497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:38.866504 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:38.866548 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:38.870604 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:38.871587 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:38.871593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:38.871597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:38.871767 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:38.872280 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:38.872286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:38.872289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:38.872825 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:38.873123 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:38.873569 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 20:57:38.873756 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:57:38.878083 31033 catalog_manager.cc:1347] Generated new cluster ID: 72ca9d09546a47fe86c10d91fe87fe8b
I20260926 20:57:38.878237 31033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:38.889395 31033 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:38.890703 31033 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:38.901129 31033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2ba9c672bbe49828f7817417ebe7b4e: Generated new TSK 0
I20260926 20:57:38.901996 31033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:38.907861 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 30941
I20260926 20:57:38.909615 31003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:39.001381 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 30941
W20260926 20:57:39.018282 30926 connection.cc:535] client connection to 127.8.249.190:46379 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.190:46379): Connection reset by peer (error 104)
2026-09-26T20:57:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1995 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:39 dist-test-slave-2dg3 krb5kdc[31040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:39 dist-test-slave-2dg3 krb5kdc[31040](info): set up 2 sockets
Sep 26 20:57:39 dist-test-slave-2dg3 krb5kdc[31040](info): commencing operation
krb5kdc: starting...
W20260926 20:57:40.119299 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:40 dist-test-slave-2dg3 krb5kdc[31040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:57:40.383504 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:44637
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:44637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:40.725543 31053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:40.725838 31053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:40.726027 31053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:40.733340 31053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:40.733446 31053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:40.733568 31053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:40.733664 31053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:40.733739 31053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:40.743687 31053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:44637
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:44637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:40.744957 31053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:40.746300 31053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:40.758025 31061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:40.758558 31059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:40.758668 31058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:40.759459 31053 server_base.cc:1034] running on GCE node
I20260926 20:57:40.760294 31053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:40.761736 31053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:40.762917 31053 hybrid_clock.cc:648] HybridClock initialized: now 1790456260762841 us; error 58 us; skew 500 ppm
I20260926 20:57:40.763281 31053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:40 dist-test-slave-2dg3 krb5kdc[31040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456260, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:40.766345 31053 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:57:40.767900 31053 webserver.cc:460] Webserver started at http://127.8.249.190:37251/ using document root <none> and password file <none>
I20260926 20:57:40.768606 31053 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:40.768767 31053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:40.769076 31053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:40.772166 31053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "8cc440b78bdd4fdd8fb2c06c4919c747"
format_stamp: "Formatted at 2026-09-26 20:57:40 on dist-test-slave-2dg3"
server_key: "75e816ce2d4d370691e3ef8583578a4d"
server_key_iv: "caec31c6fbac343bc0c2b3ce0874d3f3"
server_key_version: "encryptionkey@0"
I20260926 20:57:40.772970 31053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "8cc440b78bdd4fdd8fb2c06c4919c747"
format_stamp: "Formatted at 2026-09-26 20:57:40 on dist-test-slave-2dg3"
server_key: "75e816ce2d4d370691e3ef8583578a4d"
server_key_iv: "caec31c6fbac343bc0c2b3ce0874d3f3"
server_key_version: "encryptionkey@0"
I20260926 20:57:40.778366 31053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 20:57:40.782114 31068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:40.783697 31053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:40.783916 31053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "8cc440b78bdd4fdd8fb2c06c4919c747"
format_stamp: "Formatted at 2026-09-26 20:57:40 on dist-test-slave-2dg3"
server_key: "75e816ce2d4d370691e3ef8583578a4d"
server_key_iv: "caec31c6fbac343bc0c2b3ce0874d3f3"
server_key_version: "encryptionkey@0"
I20260926 20:57:40.784140 31053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:40.802115 31053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:40.806488 31053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:40.806823 31053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:40.826508 31053 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:44637
I20260926 20:57:40.826530 31119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:44637 every 8 connection(s)
I20260926 20:57:40.827979 31053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:40.829164 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 31053
I20260926 20:57:40.829396 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:40.829830 9190 external_mini_cluster.cc:1428] Setting key 5fc23ce407671d2cbbc9c5afa97da067
I20260926 20:57:40.832962 31120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:40.844074 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:40.831872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57409 (local address 127.8.249.190:44637)
0926 20:57:40.832681 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:40.832694 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:40.832736 (+ 42us) server_negotiation.cc:407] Connection header received
0926 20:57:40.834490 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:40.834516 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:40.835012 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:40.835383 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:40.836024 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:40.837416 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:40.839415 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:40.840123 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:40.840273 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:40.841629 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:40.841637 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:40.841654 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:40.842209 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:40.842286 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:40.842311 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:40.842444 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:40.842757 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 20:57:40.843020 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":365,"threads_started":1}
I20260926 20:57:40.845918 31120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747: Bootstrap starting.
I20260926 20:57:40.848371 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:40.849643 31120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:40.850907 31120 log.cc:824] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:40.854542 31120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747: No bootstrap required, opened a new log
I20260926 20:57:40.861249 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:40.853964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35857 (local address 127.8.249.190:44637)
0926 20:57:40.854240 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:40.854248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:40.854641 (+ 393us) server_negotiation.cc:407] Connection header received
0926 20:57:40.854841 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:40.854848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:40.854928 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:40.855026 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:40.855689 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:40.856543 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:40.858426 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:40.859010 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:40.859115 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:40.860348 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:40.860353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:40.860357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:40.860374 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:40.860436 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:40.860453 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:40.860550 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:40.860758 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 20:57:40.860843 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:57:40.864305 31120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc440b78bdd4fdd8fb2c06c4919c747" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44637 } }
I20260926 20:57:40.864679 31120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:40.864799 31120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc440b78bdd4fdd8fb2c06c4919c747, State: Initialized, Role: FOLLOWER
I20260926 20:57:40.865365 31120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc440b78bdd4fdd8fb2c06c4919c747" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44637 } }
I20260926 20:57:40.865598 31120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:40.865749 31120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:40.865938 31120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:40.868621 31120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc440b78bdd4fdd8fb2c06c4919c747" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44637 } }
I20260926 20:57:40.869143 31120 leader_election.cc:302] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cc440b78bdd4fdd8fb2c06c4919c747; no voters:
I20260926 20:57:40.869764 31120 leader_election.cc:288] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:40.869884 31130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:40.871184 31130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [term 1 LEADER]: Becoming Leader. State: Replica: 8cc440b78bdd4fdd8fb2c06c4919c747, State: Running, Role: LEADER
I20260926 20:57:40.871800 31130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc440b78bdd4fdd8fb2c06c4919c747" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44637 } }
I20260926 20:57:40.872107 31120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:40.874517 31132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cc440b78bdd4fdd8fb2c06c4919c747. Latest consensus state: current_term: 1 leader_uuid: "8cc440b78bdd4fdd8fb2c06c4919c747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc440b78bdd4fdd8fb2c06c4919c747" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44637 } } }
I20260926 20:57:40.874886 31132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:40.874634 31131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cc440b78bdd4fdd8fb2c06c4919c747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc440b78bdd4fdd8fb2c06c4919c747" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 44637 } } }
I20260926 20:57:40.875039 31131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:40.875775 31138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:40.886668 31138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:40.896844 31138 catalog_manager.cc:1347] Generated new cluster ID: efe2ffec94d14f59b578fc88805f4f64
I20260926 20:57:40.897003 31138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:40.910825 31138 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:40.912719 31138 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:40.928514 31138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cc440b78bdd4fdd8fb2c06c4919c747: Generated new TSK 0
I20260926 20:57:40.929653 31138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:40.942441 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 31053
W20260926 20:57:40.943672 31115 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57409
W20260926 20:57:40.944451 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 31053: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:57:40.944607 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 31053
W20260926 20:57:40.961311 31038 connection.cc:535] client connection to 127.8.249.190:44637 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.190:44637): Connection reset by peer (error 104)
2026-09-26T20:57:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1943 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:41 dist-test-slave-2dg3 krb5kdc[31151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:41 dist-test-slave-2dg3 krb5kdc[31151](info): set up 2 sockets
Sep 26 20:57:41 dist-test-slave-2dg3 krb5kdc[31151](info): commencing operation
krb5kdc: starting...
W20260926 20:57:42.055313 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:42 dist-test-slave-2dg3 krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:57:42.318522 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:45537
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:45537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:42.644517 31164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:42.644817 31164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:42.644992 31164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:42.652179 31164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:42.652297 31164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:42.652423 31164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:42.652546 31164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:42.652611 31164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:42.662511 31164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:45537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:45537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:42.663932 31164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:42.665302 31164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:42.677462 31172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:42.677551 31169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:42.677524 31170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:42.678287 31164 server_base.cc:1034] running on GCE node
I20260926 20:57:42.679222 31164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:42.680835 31164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:42.682061 31164 hybrid_clock.cc:648] HybridClock initialized: now 1790456262682006 us; error 60 us; skew 500 ppm
I20260926 20:57:42.682492 31164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:42 dist-test-slave-2dg3 krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456262, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:42.685777 31164 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:57:42.687341 31164 webserver.cc:460] Webserver started at http://127.8.249.190:43311/ using document root <none> and password file <none>
I20260926 20:57:42.688030 31164 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:42.688174 31164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:42.688472 31164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:42.691488 31164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "9a66bb43668e4ac78a06f3c39686bd46"
format_stamp: "Formatted at 2026-09-26 20:57:42 on dist-test-slave-2dg3"
server_key: "bd2927b3a85584ad1d5dc3758c70dd4e"
server_key_iv: "c97d92a4afeb0d98c1a8b8971d4310a4"
server_key_version: "encryptionkey@0"
I20260926 20:57:42.692266 31164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "9a66bb43668e4ac78a06f3c39686bd46"
format_stamp: "Formatted at 2026-09-26 20:57:42 on dist-test-slave-2dg3"
server_key: "bd2927b3a85584ad1d5dc3758c70dd4e"
server_key_iv: "c97d92a4afeb0d98c1a8b8971d4310a4"
server_key_version: "encryptionkey@0"
I20260926 20:57:42.697640 31164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:57:42.701154 31179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:42.702472 31164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:57:42.702692 31164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "9a66bb43668e4ac78a06f3c39686bd46"
format_stamp: "Formatted at 2026-09-26 20:57:42 on dist-test-slave-2dg3"
server_key: "bd2927b3a85584ad1d5dc3758c70dd4e"
server_key_iv: "c97d92a4afeb0d98c1a8b8971d4310a4"
server_key_version: "encryptionkey@0"
I20260926 20:57:42.703002 31164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:42.708853 31164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:42.713356 31164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:42.713660 31164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:42.734081 31164 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:45537
I20260926 20:57:42.734112 31230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:45537 every 8 connection(s)
I20260926 20:57:42.735641 31164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:42.739473 31231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:42.743804 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 31164
I20260926 20:57:42.743992 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:42.744592 9190 external_mini_cluster.cc:1428] Setting key 97030d99827fae873777e95fa65af764
I20260926 20:57:42.752549 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:42.746560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52271 (local address 127.8.249.190:45537)
0926 20:57:42.747385 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:42.747397 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:42.747438 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:57:42.749086 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:42.749110 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:42.749491 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:42.749908 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:42.751004 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:42.751009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:42.751023 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:42.751037 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:42.751088 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:42.751108 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:42.751333 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:42.751479 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 20:57:42.751666 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":381,"threads_started":1}
I20260926 20:57:42.752790 31231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46: Bootstrap starting.
I20260926 20:57:42.755903 31231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:42.756042 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:42.756940 31231 log.cc:824] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:42.759604 31231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46: No bootstrap required, opened a new log
I20260926 20:57:42.763259 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:42.760490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35757 (local address 127.8.249.190:45537)
0926 20:57:42.760661 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:42.760667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:42.761174 (+ 507us) server_negotiation.cc:407] Connection header received
0926 20:57:42.761340 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:42.761344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:42.761395 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:42.761472 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:42.762458 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:42.762462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:42.762465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:42.762480 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:42.762533 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:42.762551 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:42.762659 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:42.762901 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 20:57:42.762973 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:57:42.767874 31231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a66bb43668e4ac78a06f3c39686bd46" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 45537 } }
I20260926 20:57:42.768272 31231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:42.768380 31231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a66bb43668e4ac78a06f3c39686bd46, State: Initialized, Role: FOLLOWER
I20260926 20:57:42.768967 31231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a66bb43668e4ac78a06f3c39686bd46" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 45537 } }
I20260926 20:57:42.769217 31231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:42.769361 31231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:42.769541 31231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:42.771808 31231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a66bb43668e4ac78a06f3c39686bd46" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 45537 } }
I20260926 20:57:42.772331 31231 leader_election.cc:302] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a66bb43668e4ac78a06f3c39686bd46; no voters:
I20260926 20:57:42.773063 31231 leader_election.cc:288] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:42.773192 31241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:42.773934 31241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [term 1 LEADER]: Becoming Leader. State: Replica: 9a66bb43668e4ac78a06f3c39686bd46, State: Running, Role: LEADER
I20260926 20:57:42.774471 31241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a66bb43668e4ac78a06f3c39686bd46" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 45537 } }
I20260926 20:57:42.775267 31231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:42.778047 31242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a66bb43668e4ac78a06f3c39686bd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a66bb43668e4ac78a06f3c39686bd46" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 45537 } } }
I20260926 20:57:42.777923 31243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a66bb43668e4ac78a06f3c39686bd46. Latest consensus state: current_term: 1 leader_uuid: "9a66bb43668e4ac78a06f3c39686bd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a66bb43668e4ac78a06f3c39686bd46" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 45537 } } }
I20260926 20:57:42.778515 31243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:42.788065 31242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:42.788905 31249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:42.794173 31249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:42.803206 31249 catalog_manager.cc:1347] Generated new cluster ID: a12dfa4ee16049aa86ef0f29074092ce
I20260926 20:57:42.803342 31249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:42.816345 31249 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:42.816776 31249 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:42.822968 31249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a66bb43668e4ac78a06f3c39686bd46: Generated new TSK 0
I20260926 20:57:42.824100 31249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:42.834757 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 31164
W20260926 20:57:42.835958 31226 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52271
W20260926 20:57:42.836663 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 31164: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:57:42.836802 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 31164
2026-09-26T20:57:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:42 dist-test-slave-2dg3 krb5kdc[31262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:42 dist-test-slave-2dg3 krb5kdc[31262](info): set up 2 sockets
Sep 26 20:57:42 dist-test-slave-2dg3 krb5kdc[31262](info): commencing operation
krb5kdc: starting...
W20260926 20:57:43.939309 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:44 dist-test-slave-2dg3 krb5kdc[31262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:57:44.206177 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:45887
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:45887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:44.546443 31275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:44.546775 31275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:44.546977 31275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:44.554373 31275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:44.554507 31275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:44.554559 31275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:44.554615 31275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:44.554661 31275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:44.564996 31275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:45887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:45887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:44.566288 31275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:44.567593 31275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:44.579660 31280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:44.579720 31283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:44.579782 31281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:44.580615 31275 server_base.cc:1034] running on GCE node
I20260926 20:57:44.581425 31275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:44.583108 31275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:44.584353 31275 hybrid_clock.cc:648] HybridClock initialized: now 1790456264584274 us; error 56 us; skew 500 ppm
I20260926 20:57:44.584695 31275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:44 dist-test-slave-2dg3 krb5kdc[31262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456264, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:44.587952 31275 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:57:44.589390 31275 webserver.cc:460] Webserver started at http://127.8.249.190:34401/ using document root <none> and password file <none>
I20260926 20:57:44.590031 31275 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:44.590132 31275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:44.590385 31275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:44.593441 31275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "420d2a89d7a14ee9be5c91128f415e79"
format_stamp: "Formatted at 2026-09-26 20:57:44 on dist-test-slave-2dg3"
server_key: "2eed9d3c8c249de572a55fff7bd2d7ef"
server_key_iv: "05f3c6385a8145d224392b26dc9c59ed"
server_key_version: "encryptionkey@0"
I20260926 20:57:44.594137 31275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "420d2a89d7a14ee9be5c91128f415e79"
format_stamp: "Formatted at 2026-09-26 20:57:44 on dist-test-slave-2dg3"
server_key: "2eed9d3c8c249de572a55fff7bd2d7ef"
server_key_iv: "05f3c6385a8145d224392b26dc9c59ed"
server_key_version: "encryptionkey@0"
I20260926 20:57:44.599467 31275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 20:57:44.603519 31290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:44.605058 31275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:57:44.605293 31275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "420d2a89d7a14ee9be5c91128f415e79"
format_stamp: "Formatted at 2026-09-26 20:57:44 on dist-test-slave-2dg3"
server_key: "2eed9d3c8c249de572a55fff7bd2d7ef"
server_key_iv: "05f3c6385a8145d224392b26dc9c59ed"
server_key_version: "encryptionkey@0"
I20260926 20:57:44.605532 31275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:44.619620 31275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:44.623997 31275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:44.624310 31275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:44.642851 31275 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:45887
I20260926 20:57:44.642997 31341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:45887 every 8 connection(s)
I20260926 20:57:44.644280 31275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:44.648095 31342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:44.653539 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 31275
I20260926 20:57:44.653738 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:44.654152 9190 external_mini_cluster.cc:1428] Setting key 04c7b716a60eb7cf588f75d551f8fdc5
I20260926 20:57:44.659142 31342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79: Bootstrap starting.
I20260926 20:57:44.662984 31342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:44.664299 31342 log.cc:824] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:44.668138 31342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79: No bootstrap required, opened a new log
Sep 26 20:57:44 dist-test-slave-2dg3 krb5kdc[31262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:44.677086 31342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420d2a89d7a14ee9be5c91128f415e79" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 45887 } }
I20260926 20:57:44.677572 31342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:44.677713 31342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420d2a89d7a14ee9be5c91128f415e79, State: Initialized, Role: FOLLOWER
I20260926 20:57:44.678514 31342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420d2a89d7a14ee9be5c91128f415e79" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 45887 } }
I20260926 20:57:44.678912 31342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:44.679098 31342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:44.679328 31342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:44.680119 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:44.656122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55185 (local address 127.8.249.190:45887)
0926 20:57:44.657026 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:44.657037 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:44.657081 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:57:44.658725 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:44.658746 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:44.659248 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:44.659592 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:44.661450 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:44.662784 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:44.664721 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:44.665887 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:44.666023 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:44.671614 (+ 5591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:44.671623 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:44.671639 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:44.671680 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:44.675050 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:44.675819 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:44.675833 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:44.675839 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:44.676025 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:44.676446 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:44.676452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:44.676455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:44.677047 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:44.677368 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:44.677911 (+ 543us) server_negotiation.cc:299] Negotiation successful
0926 20:57:44.678420 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":422,"threads_started":1}
I20260926 20:57:44.682224 31342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420d2a89d7a14ee9be5c91128f415e79" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 45887 } }
I20260926 20:57:44.682966 31342 leader_election.cc:302] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 420d2a89d7a14ee9be5c91128f415e79; no voters:
I20260926 20:57:44.683724 31342 leader_election.cc:288] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:44.683799 31347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:44.684381 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:44.684511 31347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [term 1 LEADER]: Becoming Leader. State: Replica: 420d2a89d7a14ee9be5c91128f415e79, State: Running, Role: LEADER
I20260926 20:57:44.685037 31347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420d2a89d7a14ee9be5c91128f415e79" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 45887 } }
I20260926 20:57:44.686683 31347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 420d2a89d7a14ee9be5c91128f415e79. Latest consensus state: current_term: 1 leader_uuid: "420d2a89d7a14ee9be5c91128f415e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420d2a89d7a14ee9be5c91128f415e79" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 45887 } } }
I20260926 20:57:44.686977 31347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:44.687552 31349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "420d2a89d7a14ee9be5c91128f415e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420d2a89d7a14ee9be5c91128f415e79" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 45887 } } }
I20260926 20:57:44.687932 31349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:44.696987 31342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:44.697260 31351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:44.704206 31351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:44.723001 31351 catalog_manager.cc:1347] Generated new cluster ID: 70ea6ba6e07745fca5fa36a185314f48
I20260926 20:57:44.723202 31351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:44.728003 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:44.696870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35371 (local address 127.8.249.190:45887)
0926 20:57:44.697088 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:44.697096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:44.706122 (+ 9026us) server_negotiation.cc:407] Connection header received
0926 20:57:44.706340 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:44.706350 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:44.706464 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:44.707816 (+ 1352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:44.710280 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:44.711423 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:44.714346 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:44.715425 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:44.715584 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:44.720136 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:44.720149 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:44.720153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:44.720200 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:44.724686 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:44.725619 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:44.725627 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:44.725631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:44.725762 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:44.726179 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:44.726184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:44.726190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:44.726579 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:44.726851 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:44.727404 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 20:57:44.727639 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:57:44.739576 31351 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:44.741477 31351 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:44.750257 31351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 420d2a89d7a14ee9be5c91128f415e79: Generated new TSK 0
I20260926 20:57:44.751258 31351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:44.763950 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 31275
I20260926 20:57:44.765163 31337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:44.854568 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 31275
2026-09-26T20:57:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2015 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:44 dist-test-slave-2dg3 krb5kdc[31374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:44 dist-test-slave-2dg3 krb5kdc[31374](info): set up 2 sockets
Sep 26 20:57:44 dist-test-slave-2dg3 krb5kdc[31374](info): commencing operation
krb5kdc: starting...
W20260926 20:57:45.951378 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:46 dist-test-slave-2dg3 krb5kdc[31374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:57:46.228020 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36283
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:46.565414 31387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:46.565830 31387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:46.566006 31387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:46.574061 31387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:46.574263 31387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:46.574383 31387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:46.574499 31387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:46.574590 31387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:46.585250 31387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:36283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:46.586699 31387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:46.588161 31387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:46.600459 31392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:46.600394 31395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:46.600488 31393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:46.601310 31387 server_base.cc:1034] running on GCE node
I20260926 20:57:46.602175 31387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:46.603781 31387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:46.605036 31387 hybrid_clock.cc:648] HybridClock initialized: now 1790456266604990 us; error 45 us; skew 500 ppm
I20260926 20:57:46.605412 31387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:46 dist-test-slave-2dg3 krb5kdc[31374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456266, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:46.612505 31387 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:57:46.614339 31387 webserver.cc:460] Webserver started at http://127.8.249.190:37699/ using document root <none> and password file <none>
I20260926 20:57:46.615149 31387 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:46.615263 31387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:46.615535 31387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:46.618736 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "2e676a66424943e9ae4e11273d6c8b6a"
format_stamp: "Formatted at 2026-09-26 20:57:46 on dist-test-slave-2dg3"
server_key: "67d320a745bd27ec7599e517e7bb05be"
server_key_iv: "22d6e7be9221acf7f907466ce289ee0a"
server_key_version: "encryptionkey@0"
I20260926 20:57:46.619589 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "2e676a66424943e9ae4e11273d6c8b6a"
format_stamp: "Formatted at 2026-09-26 20:57:46 on dist-test-slave-2dg3"
server_key: "67d320a745bd27ec7599e517e7bb05be"
server_key_iv: "22d6e7be9221acf7f907466ce289ee0a"
server_key_version: "encryptionkey@0"
I20260926 20:57:46.625051 31387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:57:46.628708 31402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:46.630136 31387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:57:46.630285 31387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "2e676a66424943e9ae4e11273d6c8b6a"
format_stamp: "Formatted at 2026-09-26 20:57:46 on dist-test-slave-2dg3"
server_key: "67d320a745bd27ec7599e517e7bb05be"
server_key_iv: "22d6e7be9221acf7f907466ce289ee0a"
server_key_version: "encryptionkey@0"
I20260926 20:57:46.630465 31387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:46.644629 31387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:46.648936 31387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:46.649184 31387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:46.668432 31387 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:36283
I20260926 20:57:46.668439 31453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:36283 every 8 connection(s)
I20260926 20:57:46.669852 31387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:46.673576 31454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:46.674986 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 31387
I20260926 20:57:46.675161 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:46.675619 9190 external_mini_cluster.cc:1428] Setting key 4df90a8d6f970dc65fb3cf3dcd912f94
I20260926 20:57:46.687718 31454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a: Bootstrap starting.
I20260926 20:57:46.689426 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:46.677425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36889 (local address 127.8.249.190:36283)
0926 20:57:46.678230 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:46.678242 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:46.678287 (+ 45us) server_negotiation.cc:407] Connection header received
0926 20:57:46.679950 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:46.679973 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:46.680392 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:46.680725 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:46.681362 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:46.682671 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:46.684643 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:46.685389 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:46.685542 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:46.686997 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:46.687006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:46.687023 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:46.687670 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:46.687760 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:46.687786 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:46.687920 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:46.688162 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 20:57:46.688452 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":384,"threads_started":1}
I20260926 20:57:46.691114 31454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:46.692276 31454 log.cc:824] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:46.693277 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:46.695254 31454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a: No bootstrap required, opened a new log
I20260926 20:57:46.704568 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:46.697762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39325 (local address 127.8.249.190:36283)
0926 20:57:46.698043 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:46.698049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:46.698352 (+ 303us) server_negotiation.cc:407] Connection header received
0926 20:57:46.698534 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:46.698539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:46.698605 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:46.698692 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:46.699368 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:46.700123 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:46.702009 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:46.702470 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:46.702583 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:46.703814 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:46.703820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:46.703824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:46.703842 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:46.703894 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:46.703914 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:46.703992 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:46.704145 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 20:57:46.704220 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 20:57:46.705129 31454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e676a66424943e9ae4e11273d6c8b6a" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36283 } }
I20260926 20:57:46.705546 31454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:46.705668 31454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e676a66424943e9ae4e11273d6c8b6a, State: Initialized, Role: FOLLOWER
I20260926 20:57:46.706360 31454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e676a66424943e9ae4e11273d6c8b6a" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36283 } }
I20260926 20:57:46.706605 31454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:46.706763 31454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:46.706970 31454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:46.709226 31454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e676a66424943e9ae4e11273d6c8b6a" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36283 } }
I20260926 20:57:46.709755 31454 leader_election.cc:302] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e676a66424943e9ae4e11273d6c8b6a; no voters:
I20260926 20:57:46.710449 31454 leader_election.cc:288] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:46.710548 31464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:46.711308 31464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [term 1 LEADER]: Becoming Leader. State: Replica: 2e676a66424943e9ae4e11273d6c8b6a, State: Running, Role: LEADER
I20260926 20:57:46.711863 31464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e676a66424943e9ae4e11273d6c8b6a" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36283 } }
I20260926 20:57:46.712585 31454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:46.714654 31466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e676a66424943e9ae4e11273d6c8b6a. Latest consensus state: current_term: 1 leader_uuid: "2e676a66424943e9ae4e11273d6c8b6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e676a66424943e9ae4e11273d6c8b6a" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36283 } } }
I20260926 20:57:46.715039 31466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:46.717625 31470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:46.717459 31465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e676a66424943e9ae4e11273d6c8b6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e676a66424943e9ae4e11273d6c8b6a" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36283 } } }
I20260926 20:57:46.717793 31465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:46.723290 31470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:46.735352 31470 catalog_manager.cc:1347] Generated new cluster ID: cf8893280bf34d958421f81fd6aa0220
I20260926 20:57:46.735487 31470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:46.753696 31470 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:46.755622 31470 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:46.769953 31470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e676a66424943e9ae4e11273d6c8b6a: Generated new TSK 0
I20260926 20:57:46.770933 31470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:46.790107 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 31387
W20260926 20:57:46.791311 31449 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36889
W20260926 20:57:46.792060 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 31387: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:57:46.792209 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 31387
2026-09-26T20:57:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:46 dist-test-slave-2dg3 krb5kdc[31485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:46 dist-test-slave-2dg3 krb5kdc[31485](info): set up 2 sockets
Sep 26 20:57:46 dist-test-slave-2dg3 krb5kdc[31485](info): commencing operation
krb5kdc: starting...
W20260926 20:57:47.895613 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:48 dist-test-slave-2dg3 krb5kdc[31485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:57:48.162092 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:37491
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:39265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:37491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:48.504259 31498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:48.504599 31498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:48.504755 31498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:48.512141 31498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:48.512316 31498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:48.512403 31498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:48.512518 31498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:48.512588 31498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:48.522843 31498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:39265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:37491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:37491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:48.524255 31498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:48.525609 31498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:48.537401 31503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:48.537462 31504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:48.537511 31506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:48.538285 31498 server_base.cc:1034] running on GCE node
I20260926 20:57:48.539116 31498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:48.540555 31498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:48.541832 31498 hybrid_clock.cc:648] HybridClock initialized: now 1790456268541771 us; error 48 us; skew 500 ppm
I20260926 20:57:48.542248 31498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:48 dist-test-slave-2dg3 krb5kdc[31485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456268, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:48.545486 31498 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:57:48.547091 31498 webserver.cc:460] Webserver started at http://127.8.249.190:35623/ using document root <none> and password file <none>
I20260926 20:57:48.547793 31498 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:48.547942 31498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:48.548249 31498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:48.551374 31498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "73514e5dcd014b03a3a64b494bd0ffba"
format_stamp: "Formatted at 2026-09-26 20:57:48 on dist-test-slave-2dg3"
server_key: "e1b79eb555b3fec8f704f75d18f3d26b"
server_key_iv: "f2a4e6d2c1d06d7e0ec43a8761726475"
server_key_version: "encryptionkey@0"
I20260926 20:57:48.552165 31498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "73514e5dcd014b03a3a64b494bd0ffba"
format_stamp: "Formatted at 2026-09-26 20:57:48 on dist-test-slave-2dg3"
server_key: "e1b79eb555b3fec8f704f75d18f3d26b"
server_key_iv: "f2a4e6d2c1d06d7e0ec43a8761726475"
server_key_version: "encryptionkey@0"
I20260926 20:57:48.557509 31498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:57:48.561121 31513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:48.562537 31498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:57:48.562747 31498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "73514e5dcd014b03a3a64b494bd0ffba"
format_stamp: "Formatted at 2026-09-26 20:57:48 on dist-test-slave-2dg3"
server_key: "e1b79eb555b3fec8f704f75d18f3d26b"
server_key_iv: "f2a4e6d2c1d06d7e0ec43a8761726475"
server_key_version: "encryptionkey@0"
I20260926 20:57:48.563036 31498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:48.568794 31498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:48.572916 31498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:48.573242 31498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:48.592118 31498 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:37491
I20260926 20:57:48.592140 31564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:37491 every 8 connection(s)
I20260926 20:57:48.593637 31498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:48.597546 31565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:48.598531 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 31498
I20260926 20:57:48.598727 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:48.599327 9190 external_mini_cluster.cc:1428] Setting key cb9db49f7f99d4e2dd2edd7732d9f841
I20260926 20:57:48.608462 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:48.601620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37367 (local address 127.8.249.190:37491)
0926 20:57:48.602665 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:48.602681 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:48.603369 (+ 688us) server_negotiation.cc:407] Connection header received
0926 20:57:48.605073 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:48.605098 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:48.605473 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:48.605791 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:48.606930 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:48.606936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:48.606948 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:48.606963 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:48.607019 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:48.607042 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:48.607213 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:48.607318 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 20:57:48.607537 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":469,"threads_started":1}
I20260926 20:57:48.611634 31565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba: Bootstrap starting.
I20260926 20:57:48.612121 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:48.615397 31565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:48.616772 31565 log.cc:824] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:48.619958 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:48.617415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60693 (local address 127.8.249.190:37491)
0926 20:57:48.617625 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:48.617632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:48.617999 (+ 367us) server_negotiation.cc:407] Connection header received
0926 20:57:48.618203 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:48.618206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:48.618251 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:48.618356 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:48.619293 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:48.619298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:48.619303 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:57:48.619319 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:48.619364 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:57:48.619379 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:48.619493 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:48.619632 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 20:57:48.619691 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:57:48.620028 31565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba: No bootstrap required, opened a new log
I20260926 20:57:48.627772 31565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73514e5dcd014b03a3a64b494bd0ffba" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 37491 } }
I20260926 20:57:48.628190 31565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:57:48.628310 31565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73514e5dcd014b03a3a64b494bd0ffba, State: Initialized, Role: FOLLOWER
I20260926 20:57:48.628952 31565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73514e5dcd014b03a3a64b494bd0ffba" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 37491 } }
I20260926 20:57:48.629179 31565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:57:48.629325 31565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:57:48.629531 31565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:48.632021 31565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73514e5dcd014b03a3a64b494bd0ffba" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 37491 } }
I20260926 20:57:48.632455 31565 leader_election.cc:302] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73514e5dcd014b03a3a64b494bd0ffba; no voters:
I20260926 20:57:48.632973 31565 leader_election.cc:288] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:57:48.633111 31575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:48.633751 31575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [term 1 LEADER]: Becoming Leader. State: Replica: 73514e5dcd014b03a3a64b494bd0ffba, State: Running, Role: LEADER
I20260926 20:57:48.634260 31575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73514e5dcd014b03a3a64b494bd0ffba" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 37491 } }
I20260926 20:57:48.635063 31565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:48.637254 31576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73514e5dcd014b03a3a64b494bd0ffba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73514e5dcd014b03a3a64b494bd0ffba" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 37491 } } }
I20260926 20:57:48.637756 31576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:48.639505 31577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73514e5dcd014b03a3a64b494bd0ffba. Latest consensus state: current_term: 1 leader_uuid: "73514e5dcd014b03a3a64b494bd0ffba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73514e5dcd014b03a3a64b494bd0ffba" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 37491 } } }
I20260926 20:57:48.639878 31577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:48.641095 31584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:48.645124 31584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:48.653875 31584 catalog_manager.cc:1347] Generated new cluster ID: 3d7d5d73b6db445886623047ecbd336c
I20260926 20:57:48.653998 31584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:48.669946 31584 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:48.670457 31584 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:48.677652 31584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73514e5dcd014b03a3a64b494bd0ffba: Generated new TSK 0
I20260926 20:57:48.678668 31584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:48.688661 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 31498
W20260926 20:57:48.689837 31560 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37367
W20260926 20:57:48.690610 9190 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 31498: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 20:57:48.690766 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 31498
2026-09-26T20:57:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1897 ms)
[----------] 18 tests from AuthTokenIssuingTest (35181 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:48 dist-test-slave-2dg3 krb5kdc[31596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:48 dist-test-slave-2dg3 krb5kdc[31596](info): set up 2 sockets
Sep 26 20:57:48 dist-test-slave-2dg3 krb5kdc[31596](info): commencing operation
krb5kdc: starting...
W20260926 20:57:49.795490 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:49 dist-test-slave-2dg3 krb5kdc[31596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:57:50.070600 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:46839
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:46839,127.8.249.189:36255,127.8.249.188:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:50.425225 31609 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:57:50.425629 31609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:50.425736 31609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:50.425874 31609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:50.433401 31609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:50.433506 31609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:50.433563 31609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:50.433669 31609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:50.433791 31609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:50.444095 31609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:46839,127.8.249.189:36255,127.8.249.188:41949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:46839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:50.445472 31609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:50.446853 31609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:50.458936 31614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:50.459177 31617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:50.459013 31615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:50.460173 31609 server_base.cc:1034] running on GCE node
I20260926 20:57:50.460984 31609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:50.462561 31609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:50.463815 31609 hybrid_clock.cc:648] HybridClock initialized: now 1790456270463746 us; error 47 us; skew 500 ppm
I20260926 20:57:50.464195 31609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:50 dist-test-slave-2dg3 krb5kdc[31596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456270, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:50.467527 31609 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:57:50.469062 31609 webserver.cc:460] Webserver started at http://127.8.249.190:38977/ using document root <none> and password file <none>
I20260926 20:57:50.469748 31609 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:50.469838 31609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:50.470120 31609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:50.473248 31609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "335cf4efccc142aeb62181814f76dbc6"
format_stamp: "Formatted at 2026-09-26 20:57:50 on dist-test-slave-2dg3"
server_key: "2dbace269f90522913309bb4662e9d67"
server_key_iv: "672077347411145d6ac819e89339bf06"
server_key_version: "encryptionkey@0"
I20260926 20:57:50.474033 31609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "335cf4efccc142aeb62181814f76dbc6"
format_stamp: "Formatted at 2026-09-26 20:57:50 on dist-test-slave-2dg3"
server_key: "2dbace269f90522913309bb4662e9d67"
server_key_iv: "672077347411145d6ac819e89339bf06"
server_key_version: "encryptionkey@0"
I20260926 20:57:50.479472 31609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 20:57:50.483067 31624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:50.484659 31609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:57:50.484881 31609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "335cf4efccc142aeb62181814f76dbc6"
format_stamp: "Formatted at 2026-09-26 20:57:50 on dist-test-slave-2dg3"
server_key: "2dbace269f90522913309bb4662e9d67"
server_key_iv: "672077347411145d6ac819e89339bf06"
server_key_version: "encryptionkey@0"
I20260926 20:57:50.485121 31609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:50.503836 31609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:50.508461 31609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:50.508785 31609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:50.529503 31609 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:46839
I20260926 20:57:50.529510 31675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:46839 every 8 connection(s)
I20260926 20:57:50.531143 31609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:50.535457 31676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:50.539135 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 31609
I20260926 20:57:50.539361 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:50.540009 9190 external_mini_cluster.cc:1428] Setting key 0790e40cb5ba7803391ab19e4c04b74d
I20260926 20:57:50.543421 31676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:57:50 dist-test-slave-2dg3 krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456270, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.249.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.189@KRBTEST.COM" created.
I20260926 20:57:50.576072 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:50.547290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46839 (local address 127.0.0.1:39366)
0926 20:57:50.548256 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:50.548353 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:50.550986 (+ 2633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:50.552266 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:50.552272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:50.552914 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:50.553340 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:50.553363 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:50.555879 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:50.555890 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:50.557915 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:50.557936 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:50.559515 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:50.559550 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:50.559925 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:50.560834 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:50.561041 (+ 207us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:50.564876 (+ 3835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:50.570440 (+ 5564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:50.570461 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:50.570481 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:50.571414 (+ 933us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:50.571960 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:50.571967 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:50.571972 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:50.572268 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:50.573472 (+ 1204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:50.573482 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:50.573755 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 20:57:50.574075 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 20:57:50.574550 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":388,"threads_started":1}
I20260926 20:57:50.576092 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:50.548725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.8.249.190:46839)
0926 20:57:50.549495 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:50.549502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:50.549555 (+ 53us) server_negotiation.cc:407] Connection header received
0926 20:57:50.551534 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:50.551546 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:50.551944 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:50.552186 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:50.553616 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:50.555680 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:50.558130 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:50.559185 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:50.559399 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:50.565224 (+ 5825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:50.565237 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:50.565259 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:50.565308 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:50.570218 (+ 4910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:50.571589 (+ 1371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:50.571601 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:50.571605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:50.571818 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:50.572420 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:50.572425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:50.572428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:50.573288 (+ 860us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:50.573504 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:50.575065 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0926 20:57:50.575330 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":423,"threads_started":1}
I20260926 20:57:50.580192 31676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:57:50.581607 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:50.581123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:36255 (local address 127.0.0.1:56500)
0926 20:57:50.581375 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:50.581485 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260926 20:57:50.581928 31627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
I20260926 20:57:50.584334 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:50.583902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:36255 (local address 127.0.0.1:56510)
0926 20:57:50.584091 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:50.584188 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 20:57:50.584683 31676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:36255: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.189.keytab.
Entry for principal kudu/127.8.249.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.189.keytab.
I20260926 20:57:50.613560 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.189:36255
--webserver_interface=127.8.249.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.189.keytab
--principal=kudu/127.8.249.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:46839,127.8.249.189:36255,127.8.249.188:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:57:50.634122 31676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } attempt: 1
I20260926 20:57:50.635599 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:50.634979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:36255 (local address 127.0.0.1:56514)
0926 20:57:50.635234 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:50.635388 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 20:57:50.636780 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:50.636312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:36255 (local address 127.0.0.1:56520)
0926 20:57:50.636496 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:50.636594 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 20:57:50.637001 31676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:36255: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
I20260926 20:57:50.705296 31676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } attempt: 2
I20260926 20:57:50.706616 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:50.706035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:36255 (local address 127.0.0.1:56534)
0926 20:57:50.706312 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:50.706440 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 20:57:50.707871 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:50.707447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:36255 (local address 127.0.0.1:56538)
0926 20:57:50.707612 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:50.707714 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 20:57:50.708106 31676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:36255: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
I20260926 20:57:50.799391 31676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } attempt: 3
I20260926 20:57:50.800737 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:50.800136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:36255 (local address 127.0.0.1:56546)
0926 20:57:50.800378 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:50.800530 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 20:57:50.802013 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:50.801584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:36255 (local address 127.0.0.1:56556)
0926 20:57:50.801753 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:50.801840 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 20:57:50.802228 31676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:36255: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
I20260926 20:57:50.945516 31676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } attempt: 4
I20260926 20:57:50.946892 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:50.946218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:36255 (local address 127.0.0.1:56558)
0926 20:57:50.946552 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:50.946677 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260926 20:57:50.948320 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:50.947757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:36255 (local address 127.0.0.1:56560)
0926 20:57:50.947980 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:50.948101 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 20:57:50.948539 31676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:36255: Network error: Client connection negotiation failed: client connection to 127.8.249.189:36255: connect: Connection refused (error 111)
W20260926 20:57:50.955513 31683 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:57:50.955879 31683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:50.956008 31683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:50.956144 31683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:50.963814 31683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:50.963928 31683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:50.964017 31683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:50.964097 31683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:50.964206 31683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:50.974413 31683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:46839,127.8.249.189:36255,127.8.249.188:41949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.189.keytab
--principal=kudu/127.8.249.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.189:36255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.249.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:50.975778 31683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:50.977080 31683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:50.987820 31688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:50.988962 31689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:50.989949 31691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:50.990317 31683 server_base.cc:1034] running on GCE node
I20260926 20:57:50.991310 31683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:50.992792 31683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:50.994059 31683 hybrid_clock.cc:648] HybridClock initialized: now 1790456270993990 us; error 39 us; skew 500 ppm
I20260926 20:57:50.994406 31683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:50 dist-test-slave-2dg3 krb5kdc[31596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456270, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:50.997922 31683 init.cc:377] Logged in from keytab as kudu/127.8.249.189@KRBTEST.COM (short username kudu)
I20260926 20:57:50.999637 31683 webserver.cc:460] Webserver started at http://127.8.249.189:38919/ using document root <none> and password file <none>
I20260926 20:57:51.000412 31683 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:51.000550 31683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:51.000876 31683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:51.004083 31683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/data/instance:
uuid: "58d38b99225643618d373a146dbcceb0"
format_stamp: "Formatted at 2026-09-26 20:57:50 on dist-test-slave-2dg3"
server_key: "e76857f9c5f6ab5ad87abf4c00233dca"
server_key_iv: "771ac5689753f37e3c08fcb724fdf6e5"
server_key_version: "encryptionkey@0"
I20260926 20:57:51.004918 31683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/wal/instance:
uuid: "58d38b99225643618d373a146dbcceb0"
format_stamp: "Formatted at 2026-09-26 20:57:50 on dist-test-slave-2dg3"
server_key: "e76857f9c5f6ab5ad87abf4c00233dca"
server_key_iv: "771ac5689753f37e3c08fcb724fdf6e5"
server_key_version: "encryptionkey@0"
I20260926 20:57:51.010735 31683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 20:57:51.014766 31698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:51.016592 31683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:57:51.016853 31683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/wal
uuid: "58d38b99225643618d373a146dbcceb0"
format_stamp: "Formatted at 2026-09-26 20:57:50 on dist-test-slave-2dg3"
server_key: "e76857f9c5f6ab5ad87abf4c00233dca"
server_key_iv: "771ac5689753f37e3c08fcb724fdf6e5"
server_key_version: "encryptionkey@0"
I20260926 20:57:51.017143 31683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:51.044953 31683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:51.049319 31683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:51.049642 31683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:51.068650 31683 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.189:36255
I20260926 20:57:51.068670 31749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.189:36255 every 8 connection(s)
I20260926 20:57:51.070151 31683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/data/info.pb
I20260926 20:57:51.070888 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 31683
I20260926 20:57:51.071085 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-1/wal/instance
I20260926 20:57:51.071651 9190 external_mini_cluster.cc:1428] Setting key cd427dd3efdc8170f25095662a0917e0
I20260926 20:57:51.074474 31750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:51.081491 31750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.249.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.188@KRBTEST.COM" created.
Sep 26 20:57:51 dist-test-slave-2dg3 krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456270, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.189@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:51.108521 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.084395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.8.249.190:46839)
0926 20:57:51.084998 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.085006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.086116 (+ 1110us) server_negotiation.cc:407] Connection header received
0926 20:57:51.088298 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.088306 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.088408 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.088545 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.090455 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.091793 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.093523 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.094250 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.094415 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.100143 (+ 5728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.100154 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.100162 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.100231 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.104436 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.105448 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.105458 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.105465 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.105661 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.106147 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.106155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.106161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.106644 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.106863 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.107670 (+ 807us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.107963 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":293,"threads_started":1}
I20260926 20:57:51.109239 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.085045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46839 (local address 127.0.0.1:39370)
0926 20:57:51.085897 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.085949 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:51.087972 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:51.088798 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:51.088815 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:51.089488 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:51.090212 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.090234 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.092009 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.092013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.093327 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.093341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.094545 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.094551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.094770 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.095661 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:51.095687 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:51.099761 (+ 4074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:51.104685 (+ 4924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.104695 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.104711 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.105273 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.105798 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.105802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.105804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.106015 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.106846 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:51.106858 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:51.107260 (+ 402us) client_negotiation.cc:769] Sending connection context
0926 20:57:51.107517 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 20:57:51.107961 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":360,"threads_started":1}
I20260926 20:57:51.111334 31750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:57:51 dist-test-slave-2dg3 krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456270, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.189@KRBTEST.COM for kudu/127.8.249.189@KRBTEST.COM
I20260926 20:57:51.128669 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.112217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:36255 (local address 127.0.0.1:56574)
0926 20:57:51.112456 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.112477 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:51.112600 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:51.113415 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:51.113426 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:51.113964 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:51.114174 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.114189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.115746 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.115754 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.117283 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.117301 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.118221 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.118232 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.118448 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.119186 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:51.119213 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:51.122056 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:51.125841 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.125846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.125851 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.126228 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.126694 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.126698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.126702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.126909 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.127632 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:51.127640 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:51.127783 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 20:57:51.127954 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 20:57:51.128267 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 20:57:51.128837 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.112279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.8.249.189:36255)
0926 20:57:51.112959 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.112966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.112993 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:57:51.113041 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.113047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.113241 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.113369 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.114376 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.115599 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.117442 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.118026 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.118127 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.122320 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.122332 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.122347 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.122378 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.125652 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.126385 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.126396 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.126404 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.126569 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.127020 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.127027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.127031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.127505 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.127738 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.128102 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.128440 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":505,"thread_start_us":352,"threads_started":1}
I20260926 20:57:51.131590 31750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:57:51.132849 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.132269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47350)
0926 20:57:51.132484 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.132653 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.188.keytab.
Entry for principal kudu/127.8.249.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.188.keytab.
W20260926 20:57:51.133152 31702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
I20260926 20:57:51.134335 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.188:41949
--webserver_interface=127.8.249.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.188.keytab
--principal=kudu/127.8.249.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:46839,127.8.249.189:36255,127.8.249.188:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:57:51.135190 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.134491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47358)
0926 20:57:51.134896 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.135013 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235}
W20260926 20:57:51.135429 31750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:41949: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
I20260926 20:57:51.184758 31750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } attempt: 1
I20260926 20:57:51.186182 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.185597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47366)
0926 20:57:51.185905 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.186018 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 20:57:51.187518 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.187108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47370)
0926 20:57:51.187294 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.187387 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 20:57:51.187894 31750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:41949: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
I20260926 20:57:51.247836 31676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } attempt: 5
Sep 26 20:57:51 dist-test-slave-2dg3 krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456270, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for kudu/127.8.249.189@KRBTEST.COM
I20260926 20:57:51.256116 31750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } attempt: 2
I20260926 20:57:51.257202 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.256700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47376)
0926 20:57:51.256911 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.257015 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 20:57:51.258330 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.257991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47386)
0926 20:57:51.258126 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.258203 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 20:57:51.258608 31750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:41949: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
I20260926 20:57:51.263368 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.248724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.8.249.189:36255)
0926 20:57:51.248905 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.248912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.248935 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:57:51.249130 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.249135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.249219 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.249369 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.250107 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.251408 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.252620 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.253266 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.253404 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.256792 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.256799 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.256804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.256845 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.260668 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.261388 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.261395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.261398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.261535 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.261934 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.261941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.261944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.262237 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.262398 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.262826 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.263045 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:57:51.263386 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.248643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:36255 (local address 127.0.0.1:56578)
0926 20:57:51.248851 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.248879 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:51.248986 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:51.249382 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:51.249385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:51.249754 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:51.249947 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.249958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.251583 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.251587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.252464 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.252476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.253431 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.253435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.253596 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.254258 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:51.254277 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:51.256568 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:51.260840 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.260844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.260847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.261259 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.261653 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.261659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.261662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.261821 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.262370 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:51.262375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:51.262541 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 20:57:51.262685 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 20:57:51.263039 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 20:57:51.264508 31676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:57:51.265491 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.265118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47392)
0926 20:57:51.265274 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.265363 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:57:51.266362 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.266042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47396)
0926 20:57:51.266178 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.266263 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 20:57:51.266578 31676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:41949: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
I20260926 20:57:51.317835 31676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } attempt: 1
I20260926 20:57:51.319257 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.318583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47412)
0926 20:57:51.318860 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.319005 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 20:57:51.320581 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.320030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47416)
0926 20:57:51.320245 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.320381 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 20:57:51.320902 31676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:41949: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
I20260926 20:57:51.349887 31750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } attempt: 3
I20260926 20:57:51.351203 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.350666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47420)
0926 20:57:51.350956 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.351046 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 20:57:51.352352 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.351889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47432)
0926 20:57:51.352145 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.352235 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260926 20:57:51.352669 31750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:41949: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
I20260926 20:57:51.389158 31676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } attempt: 2
I20260926 20:57:51.390550 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.389929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47436)
0926 20:57:51.390183 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.390307 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 20:57:51.391791 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.391345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47452)
0926 20:57:51.391520 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.391618 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 20:57:51.392022 31676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:41949: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
W20260926 20:57:51.483465 31758 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:57:51.483800 31758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:51.483952 31758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:51.484083 31758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:51.491715 31758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:51.491811 31758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:51.491899 31758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:51.491974 31758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:51.492063 31758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:51.495903 31750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } attempt: 4
I20260926 20:57:51.497072 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.496580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47462)
0926 20:57:51.496832 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.496918 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 20:57:51.498035 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.497710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47464)
0926 20:57:51.497854 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.497933 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 20:57:51.498296 31750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:41949: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
I20260926 20:57:51.498234 31676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } attempt: 3
I20260926 20:57:51.499202 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.498826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47474)
0926 20:57:51.498986 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.499063 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 20:57:51.500167 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.499805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47480)
0926 20:57:51.500006 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.500072 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260926 20:57:51.500370 31676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:41949: Network error: Client connection negotiation failed: client connection to 127.8.249.188:41949: connect: Connection refused (error 111)
I20260926 20:57:51.502997 31758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:46839,127.8.249.189:36255,127.8.249.188:41949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.188.keytab
--principal=kudu/127.8.249.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.188:41949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.249.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:51.504316 31758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:51.505626 31758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:51.517868 31763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:51.517958 31766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:51.517928 31764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:51.518826 31758 server_base.cc:1034] running on GCE node
I20260926 20:57:51.519656 31758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:51.521265 31758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:51.522552 31758 hybrid_clock.cc:648] HybridClock initialized: now 1790456271522496 us; error 37 us; skew 500 ppm
I20260926 20:57:51.522962 31758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:51 dist-test-slave-2dg3 krb5kdc[31596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456271, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:51.526333 31758 init.cc:377] Logged in from keytab as kudu/127.8.249.188@KRBTEST.COM (short username kudu)
I20260926 20:57:51.527930 31758 webserver.cc:460] Webserver started at http://127.8.249.188:45095/ using document root <none> and password file <none>
I20260926 20:57:51.528658 31758 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:51.528782 31758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:51.529238 31758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:51.533246 31758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/data/instance:
uuid: "56400f7d310f4c91a094ee012948bcc0"
format_stamp: "Formatted at 2026-09-26 20:57:51 on dist-test-slave-2dg3"
server_key: "927f23059ce26fb6e51561f8270e0b49"
server_key_iv: "4baa423ffbb95f7ddf095eea0a612e5a"
server_key_version: "encryptionkey@0"
I20260926 20:57:51.534284 31758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/wal/instance:
uuid: "56400f7d310f4c91a094ee012948bcc0"
format_stamp: "Formatted at 2026-09-26 20:57:51 on dist-test-slave-2dg3"
server_key: "927f23059ce26fb6e51561f8270e0b49"
server_key_iv: "4baa423ffbb95f7ddf095eea0a612e5a"
server_key_version: "encryptionkey@0"
I20260926 20:57:51.540916 31758 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260926 20:57:51.544349 31773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:51.545696 31758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 20:57:51.545833 31758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/wal
uuid: "56400f7d310f4c91a094ee012948bcc0"
format_stamp: "Formatted at 2026-09-26 20:57:51 on dist-test-slave-2dg3"
server_key: "927f23059ce26fb6e51561f8270e0b49"
server_key_iv: "4baa423ffbb95f7ddf095eea0a612e5a"
server_key_version: "encryptionkey@0"
I20260926 20:57:51.546018 31758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:51.561941 31758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:51.566440 31758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:51.566661 31758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:51.585709 31758 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.188:41949
I20260926 20:57:51.585713 31824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.188:41949 every 8 connection(s)
I20260926 20:57:51.587149 31758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/data/info.pb
I20260926 20:57:51.591035 31825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:51.591517 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 31758
I20260926 20:57:51.591692 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790455916331309-9190-0/minicluster-data/master-2/wal/instance
I20260926 20:57:51.592128 9190 external_mini_cluster.cc:1428] Setting key b855092fb6c8459ccf3f4bd20d242163
I20260926 20:57:51.601467 31825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:57:51 dist-test-slave-2dg3 krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.188@KRBTEST.COM
Sep 26 20:57:51 dist-test-slave-2dg3 krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456271, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.188@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:51.621749 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.593923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.8.249.188:41949)
0926 20:57:51.594728 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.594743 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.594818 (+ 75us) server_negotiation.cc:407] Connection header received
0926 20:57:51.596458 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.596495 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.597046 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.597562 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.599026 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.600976 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.602268 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.603324 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.603502 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.608269 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.608277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.608289 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.608330 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.616625 (+ 8295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.617603 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.617613 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.617631 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.617828 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.618273 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.618281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.618285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.619087 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.619450 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.619994 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.620474 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":340,"threads_started":1}
I20260926 20:57:51.623878 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.604399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.8.249.190:46839)
0926 20:57:51.604599 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.604603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.605302 (+ 699us) server_negotiation.cc:407] Connection header received
0926 20:57:51.605532 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.605544 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.605622 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.605729 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.607008 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.607767 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.609282 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.609995 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.610136 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.618021 (+ 7885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.618026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.618029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.618062 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.620897 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.621707 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.621713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.621716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.621854 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.622288 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.622294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.622297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.622642 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.622796 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.623275 (+ 479us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.623524 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 20:57:51.623974 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.604442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:46839 (local address 127.0.0.1:39378)
0926 20:57:51.605147 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.605190 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:51.605391 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:51.605794 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:51.605801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:51.606465 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:51.606850 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.606864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.607942 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.607949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.609124 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.609136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.610252 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.610263 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.610511 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.615199 (+ 4688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:51.615225 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:51.617797 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:51.621040 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.621052 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.621059 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.621592 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.622010 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.622015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.622019 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.622176 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.622779 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:51.622787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:51.622997 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 20:57:51.623190 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 20:57:51.623524 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":362,"threads_started":1}
I20260926 20:57:51.625754 31825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:57:51 dist-test-slave-2dg3 krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456271, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.188@KRBTEST.COM for kudu/127.8.249.189@KRBTEST.COM
Sep 26 20:57:51 dist-test-slave-2dg3 krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:51.639608 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.626698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:36255 (local address 127.0.0.1:56590)
0926 20:57:51.626915 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.626936 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:51.627044 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:51.627434 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:51.627438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:51.627813 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:51.627965 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.627973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.628961 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.628969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.629863 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.629879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.630614 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.630620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.630865 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.631548 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:51.631566 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:51.633842 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:51.636998 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.637003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.637007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.637387 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.637780 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.637787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.637792 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.637989 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.638640 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:51.638644 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:51.638892 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 20:57:51.639088 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 20:57:51.639336 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 20:57:51.639837 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.626741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.8.249.189:36255)
0926 20:57:51.626972 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.626979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.626999 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:57:51.627174 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.627181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.627267 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.627386 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.628136 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.628813 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.630023 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.630455 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.630598 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.634058 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.634063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.634066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.634100 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.636843 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.637515 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.637523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.637526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.637665 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.638108 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.638111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.638113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.638450 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.638608 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.639208 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.639439 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 20:57:51.640946 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.625468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39390 (local address 127.8.249.190:46839)
0926 20:57:51.625666 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.625671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.625692 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:57:51.625874 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.625880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.625965 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.626062 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.627865 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.628573 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.629777 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.630214 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.630359 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.635027 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.635032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.635035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.635070 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.637778 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.638487 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.638495 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.638502 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.638651 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.639138 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.639143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.639145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.639501 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.640083 (+ 582us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.640367 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.640605 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:57:51.640872 31825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:57:51 dist-test-slave-2dg3 krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456271, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.188@KRBTEST.COM for kudu/127.8.249.188@KRBTEST.COM
Sep 26 20:57:51 dist-test-slave-2dg3 krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.189@KRBTEST.COM
I20260926 20:57:51.653914 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.641767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.8.249.188:41949)
0926 20:57:51.641927 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.641934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.641953 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:57:51.642019 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.642024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.642091 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.642201 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.643010 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.643686 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.644780 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.645210 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.645349 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.648462 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.648467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.648470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.648496 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.651430 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.652042 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.652047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.652051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.652163 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.652511 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.652519 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.652522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.652803 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.652970 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.653300 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.653526 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:57:51.654016 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.641600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47494)
0926 20:57:51.641794 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.641816 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:51.641918 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:51.642214 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:51.642218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:51.642635 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:51.642845 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.642857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.643802 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.643810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.644651 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.644661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.645363 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.645369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.645560 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.646156 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:51.646172 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:51.648286 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:51.651561 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.651567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.651571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.651920 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.652272 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.652277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.652280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.652411 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.652935 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:51.652940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:51.653083 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 20:57:51.653229 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 20:57:51.653674 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":137}
I20260926 20:57:51.656924 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.642855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.8.249.189:36255)
0926 20:57:51.643036 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.643042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.643061 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:57:51.643105 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.643110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.643171 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.643270 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.644431 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.645116 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.646364 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.646853 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.646991 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.651414 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.651422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.651431 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.651472 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.654205 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.654932 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.654938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.654942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.655128 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.655625 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.655629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.655631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.655910 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.656120 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.656474 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.656656 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:57:51.659850 31825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0: Bootstrap starting.
I20260926 20:57:51.662074 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:51.663790 31825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:51.665063 31825 log.cc:824] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:51.664940 31651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "335cf4efccc142aeb62181814f76dbc6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39390
I20260926 20:57:51.668079 31825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0: No bootstrap required, opened a new log
I20260926 20:57:51.675261 31825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } }
I20260926 20:57:51.675642 31825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56400f7d310f4c91a094ee012948bcc0, State: Initialized, Role: FOLLOWER
I20260926 20:57:51.676108 31825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } }
I20260926 20:57:51.677090 31831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } } }
I20260926 20:57:51.677484 31831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:57:51.677605 31676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } attempt: 4
I20260926 20:57:51.678236 31825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 20:57:51 dist-test-slave-2dg3 krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456270, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for kudu/127.8.249.188@KRBTEST.COM
W20260926 20:57:51.688328 31842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:57:51.688458 31842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:57:51.695681 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.678379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47502)
0926 20:57:51.678558 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.678577 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:51.678691 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:51.679236 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:51.679244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:51.679822 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:51.680065 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.680083 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.681523 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.681531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.683293 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.683315 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.684598 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.684604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.684813 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.685490 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:51.685508 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:51.688682 (+ 3174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:51.692773 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.692780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.692784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.693179 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.693618 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.693623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.693627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.693805 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.694533 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:51.694538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:51.694746 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 20:57:51.694927 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 20:57:51.695257 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:57:51.696177 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.678426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.8.249.188:41949)
0926 20:57:51.678934 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.678939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.678953 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:57:51.679019 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.679023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.679079 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.679156 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.680260 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.681357 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.683486 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.684229 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.684363 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.688948 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.688954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.688957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.688987 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.692628 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.693323 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.693329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.693334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.693482 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.693909 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.693913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.693933 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.694371 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.694540 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.695174 (+ 634us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.695586 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344}
I20260926 20:57:51.686022 31758 master_runner.cc:186] Error getting master registration for 127.8.249.190:46839: OK, instance_id { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" instance_seqno: 1790456270523379 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:57:51.698262 31758 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:57:51.702183 31676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6: Bootstrap starting.
I20260926 20:57:51.705617 31676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:51.706753 31676 log.cc:824] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:51.709839 31676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6: No bootstrap required, opened a new log
I20260926 20:57:51.715837 31676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } }
I20260926 20:57:51.716223 31676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 335cf4efccc142aeb62181814f76dbc6, State: Initialized, Role: FOLLOWER
I20260926 20:57:51.716696 31676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } }
I20260926 20:57:51.717701 31845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } } }
I20260926 20:57:51.718101 31845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:57:51.718993 31676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:51.719159 31651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 20:57:51.719429 31651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:51.723491 31651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } }
I20260926 20:57:51.726579 31800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "335cf4efccc142aeb62181814f76dbc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "56400f7d310f4c91a094ee012948bcc0"
I20260926 20:57:51.727165 31800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:51.730752 31800 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 335cf4efccc142aeb62181814f76dbc6 in term 1.
W20260926 20:57:51.730482 31856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:57:51.731544 31856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:57:51.732195 31628 leader_election.cc:302] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 335cf4efccc142aeb62181814f76dbc6, 56400f7d310f4c91a094ee012948bcc0; no voters:
I20260926 20:57:51.732676 31845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:51.733315 31651 leader_election.cc:288] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [CANDIDATE]: Term 1 election: Requested vote from peers 58d38b99225643618d373a146dbcceb0 (127.8.249.189:36255), 56400f7d310f4c91a094ee012948bcc0 (127.8.249.188:41949)
I20260926 20:57:51.744827 31845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [term 1 LEADER]: Becoming Leader. State: Replica: 335cf4efccc142aeb62181814f76dbc6, State: Running, Role: LEADER
I20260926 20:57:51.746095 31845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } }
W20260926 20:57:51.761914 31630 tablet.cc:2369] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:57:51.764369 31858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 335cf4efccc142aeb62181814f76dbc6. Latest consensus state: current_term: 1 leader_uuid: "335cf4efccc142aeb62181814f76dbc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } } }
I20260926 20:57:51.764847 31858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:51.767565 31864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:57:51.772085 31864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:51.768795 31609 master_runner.cc:186] Error getting master registration for 127.8.249.189:36255: OK, instance_id { permanent_uuid: "58d38b99225643618d373a146dbcceb0" instance_seqno: 1790456271063465 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:57:51.782961 31609 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:57:51.790123 31800 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0 [term 1 FOLLOWER]: Refusing update from remote peer 335cf4efccc142aeb62181814f76dbc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:57:51.791328 31858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:57:51.797489 31750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } attempt: 5
I20260926 20:57:51.818015 31831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 335cf4efccc142aeb62181814f76dbc6. Latest consensus state: current_term: 1 leader_uuid: "335cf4efccc142aeb62181814f76dbc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } } }
I20260926 20:57:51.818411 31831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:57:51.818691 31779 tablet.cc:2369] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:57:51.820684 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.789733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.8.249.190:46839)
0926 20:57:51.789936 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.789942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.794600 (+ 4658us) server_negotiation.cc:407] Connection header received
0926 20:57:51.794881 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.794888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.794991 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.795063 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.799628 (+ 4565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.800803 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.804769 (+ 3966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.805532 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.805638 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.811917 (+ 6279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.811925 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.811931 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.811993 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.815647 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.817364 (+ 1717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.817371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.817374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.817538 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.818280 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.818286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.818290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.818668 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.818969 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.820073 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.820327 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:57:51.822402 31865 mvcc.cc:205] Tried to move back new op lower bound from 7333708889239023616 to 7333708889082159104. Current Snapshot: MvccSnapshot[applied={T|T < 7333708889239023616}]
I20260926 20:57:51.825222 31868 mvcc.cc:205] Tried to move back new op lower bound from 7333708889239023616 to 7333708889082159104. Current Snapshot: MvccSnapshot[applied={T|T < 7333708889239023616}]
I20260926 20:57:51.834538 31845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "335cf4efccc142aeb62181814f76dbc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } } }
I20260926 20:57:51.834895 31845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:51.834885 31831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "335cf4efccc142aeb62181814f76dbc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } } }
I20260926 20:57:51.835140 31831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:57:51.836463 31864 catalog_manager.cc:1347] Generated new cluster ID: 2124daa206074d6e8b5ca06358673a99
I20260926 20:57:51.836557 31864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:51.837102 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.797720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.8.249.188:41949)
0926 20:57:51.797931 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.797939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.815377 (+ 17438us) server_negotiation.cc:407] Connection header received
0926 20:57:51.815662 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.815668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.815766 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.815835 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.818120 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.819262 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.821025 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.821685 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.821821 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.825645 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.825655 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.825662 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.825714 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.830605 (+ 4891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.831698 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.831703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.831706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.831968 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.832642 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.832648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.832651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.833321 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.834553 (+ 1232us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.835045 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.835941 (+ 896us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 20:57:51.851701 31864 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:51.853790 31864 catalog_manager.cc:1504] Loading token signing keys...
Sep 26 20:57:51 dist-test-slave-2dg3 krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456270, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.189@KRBTEST.COM for kudu/127.8.249.188@KRBTEST.COM
I20260926 20:57:51.865121 31864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6: Generated new TSK 0
I20260926 20:57:51.866323 31864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:51.868343 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.798147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:41949 (local address 127.0.0.1:47520)
0926 20:57:51.798878 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:51.798902 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:51.799028 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:51.838017 (+ 38989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:51.838023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:51.838670 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:51.838940 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.838955 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.841497 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.841503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.842992 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:51.843009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.852684 (+ 9675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.852690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:51.852963 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.853572 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:51.853595 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:51.856486 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:51.864224 (+ 7738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.864229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.864233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.864757 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.865709 (+ 952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:51.865713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:51.865717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:51.865897 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:51.867366 (+ 1469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:51.867372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:51.867546 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 20:57:51.867705 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 20:57:51.868026 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537}
I20260926 20:57:51.869338 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.799351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.8.249.188:41949)
0926 20:57:51.837465 (+ 38114us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.837470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.837516 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:57:51.837601 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.837606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.837731 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.839122 (+ 1391us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.839383 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.841103 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.843692 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.852290 (+ 8598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.852586 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.856769 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.856777 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.856815 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.856888 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.864027 (+ 7139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.865140 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.865146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.865150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.865527 (+ 377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.866071 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.866077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.866081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.867147 (+ 1066us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.867451 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.867980 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.868483 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37931,"thread_start_us":52342,"threads_started":1}
I20260926 20:57:51.871687 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:51.786302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.8.249.189:36255)
0926 20:57:51.786501 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:51.786507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:51.794339 (+ 7832us) server_negotiation.cc:407] Connection header received
0926 20:57:51.794783 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:51.794807 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:51.794909 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:51.794988 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:51.814257 (+ 19269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.815660 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.846758 (+ 31098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:51.847439 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:51.847580 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:51.860446 (+ 12866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:51.860453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:51.860457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:51.860498 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:51.863987 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.868711 (+ 4724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.868716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.868720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.868880 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:51.869295 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:51.869303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:51.869306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:51.869880 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:51.870120 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:51.871040 (+ 920us) server_negotiation.cc:299] Negotiation successful
0926 20:57:51.871284 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 20:57:51.874254 31750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0: Bootstrap starting.
I20260926 20:57:51.877507 31750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:51.879015 31750 log.cc:824] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:51.882823 31750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0: No bootstrap required, opened a new log
I20260926 20:57:51.907944 31750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } }
I20260926 20:57:51.908502 31750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d38b99225643618d373a146dbcceb0, State: Initialized, Role: FOLLOWER
I20260926 20:57:51.909125 31750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } }
I20260926 20:57:51.916239 31876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } } }
I20260926 20:57:51.918656 31876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:57:51.918578 31725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "335cf4efccc142aeb62181814f76dbc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "58d38b99225643618d373a146dbcceb0"
I20260926 20:57:51.918515 31750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:51.919148 31725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:51.923744 31725 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 335cf4efccc142aeb62181814f76dbc6 in term 1.
I20260926 20:57:51.924633 31724 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0 [term 1 FOLLOWER]: Refusing update from remote peer 335cf4efccc142aeb62181814f76dbc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:57:51.927088 31845 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:57:51.970651 31876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 335cf4efccc142aeb62181814f76dbc6. Latest consensus state: current_term: 1 leader_uuid: "335cf4efccc142aeb62181814f76dbc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } } }
I20260926 20:57:51.971052 31876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:57:51.971436 31885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "335cf4efccc142aeb62181814f76dbc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } } }
I20260926 20:57:51.972030 31885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:57:51.995342 9190 mini_kdc.cc:348] Time spent kdestroy: real 0.107s user 0.000s sys 0.022s
I20260926 20:57:52.017660 31893 catalog_manager.cc:1259] Loaded cluster ID: 2124daa206074d6e8b5ca06358673a99
I20260926 20:57:52.017833 31893 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0: loading cluster ID for follower catalog manager: success
I20260926 20:57:52.032110 31893 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0: acquiring CA information for follower catalog manager: success
I20260926 20:57:52.035712 31893 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 58d38b99225643618d373a146dbcceb0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 20:57:52.038507 31756 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:52.030584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.8.249.189:36255)
0926 20:57:52.030869 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.030878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.036542 (+ 5664us) server_negotiation.cc:407] Connection header received
0926 20:57:52.036892 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.036899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.037007 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.037089 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:52.038243 (+ 1154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56624: BlockingRecv error: recv got EOF from 127.0.0.1:56624 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 20:57:52.039640 31753 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:52.034206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.8.249.190:46839)
0926 20:57:52.034425 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.034431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.036907 (+ 2476us) server_negotiation.cc:407] Connection header received
0926 20:57:52.038061 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.038067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.038150 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.038221 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:52.039354 (+ 1133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39420: BlockingRecv error: recv got EOF from 127.0.0.1:39420 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 20:57:52.041046 31828 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:52.037422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.8.249.188:41949)
0926 20:57:52.037626 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.037632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.039357 (+ 1725us) server_negotiation.cc:407] Connection header received
0926 20:57:52.039496 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.039502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.039602 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.039715 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:52.040814 (+ 1099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47534: BlockingRecv error: recv got EOF from 127.0.0.1:47534 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:57:52.117403 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:52.101598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.8.249.189:36255)
0926 20:57:52.101794 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.101800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.109511 (+ 7711us) server_negotiation.cc:407] Connection header received
0926 20:57:52.110012 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.110018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.110206 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.110338 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:52.111143 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.112327 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.114435 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.115164 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.115264 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:52.115956 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:52.116607 (+ 651us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:52.116731 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:52.116958 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 20:57:52.117048 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 20:57:52.164058 31828 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:52.155081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47546 (local address 127.8.249.188:41949)
0926 20:57:52.155303 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.155309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.162137 (+ 6828us) server_negotiation.cc:407] Connection header received
0926 20:57:52.162447 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.162461 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.162656 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.162815 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:52.163767 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47546: BlockingRecv error: recv got EOF from 127.0.0.1:47546 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:57:52.212728 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:52.197903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.8.249.189:36255)
0926 20:57:52.198100 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.198112 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.205589 (+ 7477us) server_negotiation.cc:407] Connection header received
0926 20:57:52.206110 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.206117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.206239 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.206324 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:52.207018 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.208186 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.210314 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.211065 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.211170 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:52.211816 (+ 646us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:52.212001 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:52.212068 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:52.212309 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 20:57:52.212398 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 20:57:52.256456 31828 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:52.247090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47558 (local address 127.8.249.188:41949)
0926 20:57:52.247302 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.247308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.254367 (+ 7059us) server_negotiation.cc:407] Connection header received
0926 20:57:52.254969 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.254976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.255074 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.255149 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:52.256190 (+ 1041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47558: BlockingRecv error: recv got EOF from 127.0.0.1:47558 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:57:52.314018 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:52.295315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.8.249.189:36255)
0926 20:57:52.295565 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.295573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.304488 (+ 8915us) server_negotiation.cc:407] Connection header received
0926 20:57:52.304694 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.304700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.304795 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.304899 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:52.305544 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.306777 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.311619 (+ 4842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.312302 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.312451 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:52.313096 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:52.313267 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:52.313336 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:52.313572 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 20:57:52.313679 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260926 20:57:52.335031 31828 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:52.332475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.8.249.188:41949)
0926 20:57:52.332737 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.332744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.333364 (+ 620us) server_negotiation.cc:407] Connection header received
0926 20:57:52.333582 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.333588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.333691 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.333808 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:52.334731 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47566: BlockingRecv error: recv got EOF from 127.0.0.1:47566 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 20:57:52.352499 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:52.345343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.8.249.189:36255)
0926 20:57:52.345548 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.345556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.345681 (+ 125us) server_negotiation.cc:407] Connection header received
0926 20:57:52.346013 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.346020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.346122 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.346236 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:52.346760 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.348087 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.350085 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.350901 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.351058 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:52.351524 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:52.351676 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:52.351772 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:52.351936 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 20:57:52.352026 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 20:57:52.364902 31828 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:52.360514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.8.249.188:41949)
0926 20:57:52.360790 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.360796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.363365 (+ 2569us) server_negotiation.cc:407] Connection header received
0926 20:57:52.363581 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.363587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.363693 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.363816 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:52.364672 (+ 856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47568: BlockingRecv error: recv got EOF from 127.0.0.1:47568 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
I20260926 20:57:52.387465 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:52.380129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.8.249.189:36255)
0926 20:57:52.380355 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.380361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.380947 (+ 586us) server_negotiation.cc:407] Connection header received
0926 20:57:52.381182 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.381189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.381279 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.381397 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:52.381939 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.383143 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.385075 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.385725 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.385881 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:52.386499 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:52.386683 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:52.386822 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:52.387014 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 20:57:52.387106 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 20:57:52.399379 31828 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:52.397033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.8.249.188:41949)
0926 20:57:52.397223 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.397230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.397748 (+ 518us) server_negotiation.cc:407] Connection header received
0926 20:57:52.397970 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.397976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.398086 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.398223 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:52.399154 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47570: BlockingRecv error: recv got EOF from 127.0.0.1:47570 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:57:52.426738 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:52.419080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.8.249.189:36255)
0926 20:57:52.419335 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.419342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.419967 (+ 625us) server_negotiation.cc:407] Connection header received
0926 20:57:52.420281 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.420287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.420390 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.420512 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:52.421119 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.422361 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.424418 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.425122 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.425266 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:52.425926 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:52.426077 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:52.426169 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:52.426353 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 20:57:52.426439 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 20:57:52.436483 31828 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:52.434208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.8.249.188:41949)
0926 20:57:52.434407 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.434413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.434886 (+ 473us) server_negotiation.cc:407] Connection header received
0926 20:57:52.435101 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.435107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.435200 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.435323 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:52.436235 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47584: BlockingRecv error: recv got EOF from 127.0.0.1:47584 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:57:52.480387 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:52.473076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.8.249.189:36255)
0926 20:57:52.473305 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.473312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.473883 (+ 571us) server_negotiation.cc:407] Connection header received
0926 20:57:52.474112 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.474119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.474218 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.474336 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:52.474847 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.475939 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.477916 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.478685 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.478807 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:52.479471 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:52.479626 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:52.479715 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:52.479968 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 20:57:52.480054 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 20:57:52.496069 31845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "335cf4efccc142aeb62181814f76dbc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "335cf4efccc142aeb62181814f76dbc6" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 46839 } } peers { permanent_uuid: "58d38b99225643618d373a146dbcceb0" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 36255 } } peers { permanent_uuid: "56400f7d310f4c91a094ee012948bcc0" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 41949 } } }
I20260926 20:57:52.498834 31845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 335cf4efccc142aeb62181814f76dbc6 [sys.catalog]: This master's current role is: LEADER
W20260926 20:57:52.509521 31828 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:52.499610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.8.249.188:41949)
0926 20:57:52.499800 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.499806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.508039 (+ 8233us) server_negotiation.cc:407] Connection header received
0926 20:57:52.508217 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.508223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.508322 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.508437 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:52.509278 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47590: BlockingRecv error: recv got EOF from 127.0.0.1:47590 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:57:52.582024 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:52.576902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.8.249.189:36255)
0926 20:57:52.577181 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.577188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.577417 (+ 229us) server_negotiation.cc:407] Connection header received
0926 20:57:52.577575 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.577579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.577654 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.577736 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:52.578176 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.578943 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.580250 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.580646 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.580761 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:52.581153 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:52.581321 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:52.581409 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:52.581611 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 20:57:52.581691 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260926 20:57:52.596937 31828 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:52.587764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.8.249.188:41949)
0926 20:57:52.587931 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.587936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.595438 (+ 7502us) server_negotiation.cc:407] Connection header received
0926 20:57:52.595679 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.595683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.595761 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.595876 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:52.596706 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47596: BlockingRecv error: recv got EOF from 127.0.0.1:47596 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:57:52.690846 31842 catalog_manager.cc:1259] Loaded cluster ID: 2124daa206074d6e8b5ca06358673a99
I20260926 20:57:52.690963 31842 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0: loading cluster ID for follower catalog manager: success
I20260926 20:57:52.693534 31842 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0: acquiring CA information for follower catalog manager: success
I20260926 20:57:52.695448 31842 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 56400f7d310f4c91a094ee012948bcc0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:57:52.734716 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:52.729031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.8.249.189:36255)
0926 20:57:52.729287 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.729294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.729633 (+ 339us) server_negotiation.cc:407] Connection header received
0926 20:57:52.729881 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.729886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.729969 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.730077 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:52.730533 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.731559 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.733018 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.733434 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.733576 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:52.733982 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:52.734087 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:52.734187 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:52.734362 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 20:57:52.734444 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 20:57:52.746080 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:52.740525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.8.249.188:41949)
0926 20:57:52.740792 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.740799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.741038 (+ 239us) server_negotiation.cc:407] Connection header received
0926 20:57:52.741203 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.741208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.741381 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.741495 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:52.741859 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.742622 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.743976 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.744408 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.744519 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:52.744937 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:52.745477 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:52.745589 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:52.745656 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 20:57:52.745728 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 20:57:52.762372 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:52.752942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39430 (local address 127.8.249.190:46839)
0926 20:57:52.755250 (+ 2308us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.755258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.755277 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:57:52.755423 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.755432 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.755663 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.756618 (+ 955us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:52.756796 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.757863 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.760158 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.760669 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.760830 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:52.761334 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:52.761676 (+ 342us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:52.761811 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:52.761921 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 20:57:52.762011 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2167,"thread_start_us":6182,"threads_started":1}
I20260926 20:57:52.763329 31756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:52.753105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.8.249.189:36255)
0926 20:57:52.753360 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.753370 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.755286 (+ 1916us) server_negotiation.cc:407] Connection header received
0926 20:57:52.755562 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.755568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.755650 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.755720 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:52.756628 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.757789 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.760942 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.761625 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.761722 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:52.762509 (+ 787us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:52.762648 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:52.762735 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:52.763000 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 20:57:52.763086 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 20:57:52.766206 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:52.754186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.8.249.188:41949)
0926 20:57:52.754369 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:52.754377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:52.759760 (+ 5383us) server_negotiation.cc:407] Connection header received
0926 20:57:52.759980 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:52.759985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:52.760091 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:52.760187 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:52.760918 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.761973 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.764416 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:52.764950 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:52.765047 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:52.765504 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:52.765604 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:52.765708 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:52.765837 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 20:57:52.765924 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:57:52.769369 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 31609
I20260926 20:57:52.770507 31671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:52.881094 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 31609
I20260926 20:57:52.902505 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 31683
I20260926 20:57:52.903688 31745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:53.010908 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 31683
I20260926 20:57:53.031597 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 31758
I20260926 20:57:53.032737 31820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:53.139384 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 31758
2026-09-26T20:57:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4454 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:57:53 dist-test-slave-2dg3 krb5kdc[32008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:57:53 dist-test-slave-2dg3 krb5kdc[32008](info): set up 2 sockets
Sep 26 20:57:53 dist-test-slave-2dg3 krb5kdc[32008](info): commencing operation
krb5kdc: starting...
W20260926 20:57:54.255363 9190 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/test-user.keytab.
Sep 26 20:57:54 dist-test-slave-2dg3 krb5kdc[32008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:57:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:57:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
Entry for principal kudu/127.8.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab.
I20260926 20:57:54.526528 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36141
--webserver_interface=127.8.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:36141,127.8.249.189:44175,127.8.249.188:33291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:57:54.860898 32021 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:57:54.861279 32021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:54.861414 32021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:54.861531 32021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:54.868916 32021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:54.869021 32021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:54.869078 32021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:54.869122 32021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:54.869243 32021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:54.879436 32021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:36141,127.8.249.189:44175,127.8.249.188:33291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.190.keytab
--principal=kudu/127.8.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.190:36141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:54.880865 32021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:54.882282 32021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:54.894284 32026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:54.894369 32027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:54.894327 32029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:54.895433 32021 server_base.cc:1034] running on GCE node
I20260926 20:57:54.896235 32021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:54.897652 32021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:54.898931 32021 hybrid_clock.cc:648] HybridClock initialized: now 1790456274898854 us; error 42 us; skew 500 ppm
I20260926 20:57:54.899278 32021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:54 dist-test-slave-2dg3 krb5kdc[32008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456274, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:54.902329 32021 init.cc:377] Logged in from keytab as kudu/127.8.249.190@KRBTEST.COM (short username kudu)
I20260926 20:57:54.903855 32021 webserver.cc:460] Webserver started at http://127.8.249.190:41079/ using document root <none> and password file <none>
I20260926 20:57:54.904539 32021 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:54.904670 32021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:54.904996 32021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:54.907991 32021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/data/instance:
uuid: "089dc8cc023747bdbc16668bad02715c"
format_stamp: "Formatted at 2026-09-26 20:57:54 on dist-test-slave-2dg3"
server_key: "35d948fb3bb19c49ef44627bfd8e2690"
server_key_iv: "cf83369bea6c37334345529dbe993d5c"
server_key_version: "encryptionkey@0"
I20260926 20:57:54.908759 32021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/wal/instance:
uuid: "089dc8cc023747bdbc16668bad02715c"
format_stamp: "Formatted at 2026-09-26 20:57:54 on dist-test-slave-2dg3"
server_key: "35d948fb3bb19c49ef44627bfd8e2690"
server_key_iv: "cf83369bea6c37334345529dbe993d5c"
server_key_version: "encryptionkey@0"
I20260926 20:57:54.913785 32021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:57:54.917150 32036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:54.918476 32021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 20:57:54.918669 32021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/wal
uuid: "089dc8cc023747bdbc16668bad02715c"
format_stamp: "Formatted at 2026-09-26 20:57:54 on dist-test-slave-2dg3"
server_key: "35d948fb3bb19c49ef44627bfd8e2690"
server_key_iv: "cf83369bea6c37334345529dbe993d5c"
server_key_version: "encryptionkey@0"
I20260926 20:57:54.918952 32021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:54.952157 32021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:54.956362 32021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:54.956653 32021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:54.975991 32021 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.190:36141
I20260926 20:57:54.976017 32087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.190:36141 every 8 connection(s)
I20260926 20:57:54.977463 32021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/data/info.pb
I20260926 20:57:54.981474 32088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:54.983343 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 32021
I20260926 20:57:54.983538 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-0/wal/instance
I20260926 20:57:54.983966 9190 external_mini_cluster.cc:1428] Setting key 1ff362d1119bb663c56e4851d7a40cba
I20260926 20:57:54.988905 32088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:57:54 dist-test-slave-2dg3 krb5kdc[32008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456274, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.249.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.189@KRBTEST.COM" created.
I20260926 20:57:55.014357 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:54.993851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.8.249.190:36141)
0926 20:57:54.994660 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:54.994670 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:54.994734 (+ 64us) server_negotiation.cc:407] Connection header received
0926 20:57:54.996223 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:54.996233 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:54.996604 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:54.996838 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:54.998128 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.000125 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.001566 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.002570 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.002818 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:55.006756 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:55.006765 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:55.006779 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:55.006977 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:55.010052 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:55.010847 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:55.010854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:55.010856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:55.010997 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:55.011391 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:55.011399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:55.011402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:55.012095 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:55.012307 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:55.012915 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 20:57:55.013372 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":192,"server-negotiator.queue_time_us":580,"thread_start_us":394,"threads_started":1}
I20260926 20:57:55.014330 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:54.992397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36141 (local address 127.0.0.1:34420)
0926 20:57:54.993331 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:54.993393 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:54.995802 (+ 2409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:54.996901 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:54.996907 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:54.997523 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:54.997897 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:54.997916 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.000307 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.000311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:55.001408 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:55.001421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.002896 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.002904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:55.003083 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:55.003790 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:55.003813 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:55.006460 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:55.010206 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:55.010217 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:55.010229 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:55.010616 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:55.011109 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:55.011116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:55.011119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:55.011282 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:55.012269 (+ 987us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:55.012276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:55.012478 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 20:57:55.012741 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 20:57:55.013178 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":1,"thread_start_us":409,"threads_started":1}
I20260926 20:57:55.018013 32088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:57:55.019348 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.018906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:44175 (local address 127.0.0.1:43866)
0926 20:57:55.019089 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.019216 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 20:57:55.019716 32038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
I20260926 20:57:55.021564 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.021091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:44175 (local address 127.0.0.1:43880)
0926 20:57:55.021269 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.021367 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 20:57:55.021816 32088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:44175: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.189.keytab.
Entry for principal kudu/127.8.249.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.189.keytab.
I20260926 20:57:55.046047 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.189:44175
--webserver_interface=127.8.249.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.189.keytab
--principal=kudu/127.8.249.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:36141,127.8.249.189:44175,127.8.249.188:33291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:57:55.071192 32088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } attempt: 1
I20260926 20:57:55.072458 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.071985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:44175 (local address 127.0.0.1:43892)
0926 20:57:55.072185 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.072309 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 20:57:55.073539 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.073150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:44175 (local address 127.0.0.1:43900)
0926 20:57:55.073312 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.073397 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 20:57:55.073832 32088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:44175: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
I20260926 20:57:55.142108 32088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } attempt: 2
I20260926 20:57:55.143392 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.142875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:44175 (local address 127.0.0.1:43906)
0926 20:57:55.143133 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.143220 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 20:57:55.144763 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.144252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:44175 (local address 127.0.0.1:43916)
0926 20:57:55.144458 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.144581 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 20:57:55.144996 32088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:44175: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
I20260926 20:57:55.236326 32088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } attempt: 3
I20260926 20:57:55.237789 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.237170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:44175 (local address 127.0.0.1:43920)
0926 20:57:55.237460 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.237602 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 20:57:55.239146 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.238627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:44175 (local address 127.0.0.1:43932)
0926 20:57:55.238831 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.238937 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 20:57:55.239454 32088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:44175: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
I20260926 20:57:55.382762 32088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } attempt: 4
I20260926 20:57:55.384146 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.383477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:44175 (local address 127.0.0.1:43942)
0926 20:57:55.383842 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.383963 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260926 20:57:55.385273 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.384827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:44175 (local address 127.0.0.1:43950)
0926 20:57:55.384997 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.385102 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 20:57:55.385463 32088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.189:44175: Network error: Client connection negotiation failed: client connection to 127.8.249.189:44175: connect: Connection refused (error 111)
W20260926 20:57:55.399283 32095 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:57:55.399626 32095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:55.399719 32095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:55.399811 32095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:57:55.407467 32095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:55.407589 32095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:55.407655 32095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:55.407722 32095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:55.407795 32095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:55.419147 32095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:36141,127.8.249.189:44175,127.8.249.188:33291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.189.keytab
--principal=kudu/127.8.249.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.189:44175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.249.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:55.420481 32095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:55.421852 32095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:55.434876 32101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:55.434890 32100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:55.434902 32103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:55.435987 32095 server_base.cc:1034] running on GCE node
I20260926 20:57:55.436787 32095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:55.438344 32095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:55.439590 32095 hybrid_clock.cc:648] HybridClock initialized: now 1790456275439515 us; error 48 us; skew 500 ppm
I20260926 20:57:55.439940 32095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:55 dist-test-slave-2dg3 krb5kdc[32008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456275, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:55.443110 32095 init.cc:377] Logged in from keytab as kudu/127.8.249.189@KRBTEST.COM (short username kudu)
I20260926 20:57:55.444482 32095 webserver.cc:460] Webserver started at http://127.8.249.189:35135/ using document root <none> and password file <none>
I20260926 20:57:55.445176 32095 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:55.445305 32095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:55.445626 32095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:55.448578 32095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/data/instance:
uuid: "85173e5e4ca54fd9bf4cdd5896e47660"
format_stamp: "Formatted at 2026-09-26 20:57:55 on dist-test-slave-2dg3"
server_key: "150c33e47af054d468e051dafd5002ab"
server_key_iv: "f4fcade90b575dd9c459d1a30d13dd9c"
server_key_version: "encryptionkey@0"
I20260926 20:57:55.449388 32095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/wal/instance:
uuid: "85173e5e4ca54fd9bf4cdd5896e47660"
format_stamp: "Formatted at 2026-09-26 20:57:55 on dist-test-slave-2dg3"
server_key: "150c33e47af054d468e051dafd5002ab"
server_key_iv: "f4fcade90b575dd9c459d1a30d13dd9c"
server_key_version: "encryptionkey@0"
I20260926 20:57:55.454706 32095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:57:55.458389 32110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:55.459910 32095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 20:57:55.460124 32095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/wal
uuid: "85173e5e4ca54fd9bf4cdd5896e47660"
format_stamp: "Formatted at 2026-09-26 20:57:55 on dist-test-slave-2dg3"
server_key: "150c33e47af054d468e051dafd5002ab"
server_key_iv: "f4fcade90b575dd9c459d1a30d13dd9c"
server_key_version: "encryptionkey@0"
I20260926 20:57:55.460363 32095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:55.495342 32095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:55.499621 32095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:55.499917 32095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:55.518409 32095 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.189:44175
I20260926 20:57:55.518427 32161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.189:44175 every 8 connection(s)
I20260926 20:57:55.519829 32095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/data/info.pb
I20260926 20:57:55.522884 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 32095
I20260926 20:57:55.523087 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-1/wal/instance
I20260926 20:57:55.523535 9190 external_mini_cluster.cc:1428] Setting key 3f2619ce50da7efe42ca7bf0d77a2881
I20260926 20:57:55.523945 32162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:55.530351 32162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:57:55 dist-test-slave-2dg3 krb5kdc[32008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456275, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.189@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.249.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.249.188@KRBTEST.COM" created.
I20260926 20:57:55.555135 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.533171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.8.249.190:36141)
0926 20:57:55.533695 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:55.533701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:55.534670 (+ 969us) server_negotiation.cc:407] Connection header received
0926 20:57:55.537575 (+ 2905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:55.537580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:55.537657 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:55.537766 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:55.539965 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.540687 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.543154 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.543569 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.543664 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:55.548445 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:55.548450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:55.548454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:55.548491 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:55.551455 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:55.552431 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:55.552440 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:55.552447 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:55.552574 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:55.553118 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:55.553122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:55.553124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:55.553432 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:55.553559 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:55.554597 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0926 20:57:55.554847 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":258,"threads_started":1}
I20260926 20:57:55.556700 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.533673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36141 (local address 127.0.0.1:34432)
0926 20:57:55.534484 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.534537 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:55.537258 (+ 2721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:55.538044 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:55.538057 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:55.538937 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:55.539717 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:55.539756 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.540865 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.540873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:55.543008 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:55.543026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.543810 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.543819 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:55.544156 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:55.544926 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:55.544961 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:55.548146 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:55.551632 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:55.551641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:55.551657 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:55.552265 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:55.552698 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:55.552703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:55.552707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:55.552961 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:55.553590 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:55.553601 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:55.554144 (+ 543us) client_negotiation.cc:769] Sending connection context
0926 20:57:55.554492 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 20:57:55.555171 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":301,"threads_started":1}
I20260926 20:57:55.558614 32162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:57:55 dist-test-slave-2dg3 krb5kdc[32008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456275, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.189@KRBTEST.COM for kudu/127.8.249.189@KRBTEST.COM
I20260926 20:57:55.573369 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.559516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:44175 (local address 127.0.0.1:43966)
0926 20:57:55.559688 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.559710 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:55.559814 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:55.560733 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:55.560737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:55.561190 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:55.561335 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:55.561344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.562770 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.562777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:55.563656 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:55.563667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.564540 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.564545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:55.564734 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:55.565349 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:55.565367 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:55.567457 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:55.570697 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:55.570702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:55.570707 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:55.571135 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:55.571532 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:55.571537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:55.571540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:55.571673 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:55.572289 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:55.572294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:55.572439 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 20:57:55.572587 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 20:57:55.572890 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":1}
I20260926 20:57:55.573414 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.559674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.8.249.189:44175)
0926 20:57:55.560333 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:55.560339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:55.560362 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:57:55.560405 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:55.560411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:55.560588 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:55.560753 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:55.561515 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.562640 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.563822 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.564399 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.564526 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:55.567677 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:55.567688 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:55.567702 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:55.567729 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:55.570543 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:55.571270 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:55.571283 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:55.571285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:55.571400 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:55.571790 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:55.571795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:55.571803 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:55.572151 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:55.572380 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:55.572735 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 20:57:55.573020 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":512,"thread_start_us":386,"threads_started":1}
I20260926 20:57:55.576272 32162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:57:55.577468 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.576991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53330)
0926 20:57:55.577165 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.577311 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 20:57:55.577780 32114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
I20260926 20:57:55.579289 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.578959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53342)
0926 20:57:55.579101 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.579180 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:57:55.579524 32162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:33291: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.249.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.188.keytab.
Entry for principal kudu/127.8.249.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.188.keytab.
I20260926 20:57:55.586983 9190 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
/tmp/dist-test-taskpGD2b0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.188:33291
--webserver_interface=127.8.249.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.188.keytab
--principal=kudu/127.8.249.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.249.148:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.249.190:36141,127.8.249.189:44175,127.8.249.188:33291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:57:55.628916 32162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } attempt: 1
I20260926 20:57:55.630101 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.629662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53358)
0926 20:57:55.629855 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.629954 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 20:57:55.631251 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.630885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53370)
0926 20:57:55.631024 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.631095 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 20:57:55.631553 32162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:33291: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
I20260926 20:57:55.684759 32088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } attempt: 5
Sep 26 20:57:55 dist-test-slave-2dg3 krb5kdc[32008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456274, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for kudu/127.8.249.189@KRBTEST.COM
I20260926 20:57:55.698961 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.685611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:44175 (local address 127.0.0.1:43982)
0926 20:57:55.685854 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.685877 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:55.686035 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:55.686418 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:55.686422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:55.686928 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:55.687105 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:55.687113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.688089 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.688097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:55.688964 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:55.688975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.689747 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.689753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:55.689974 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:55.690636 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:55.690654 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:55.692881 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:55.696499 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:55.696502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:55.696505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:55.696862 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:55.697271 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:55.697275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:55.697277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:55.697483 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:55.698070 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:55.698073 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:55.698224 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 20:57:55.698375 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 20:57:55.698588 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 20:57:55.699028 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.685684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.8.249.189:44175)
0926 20:57:55.685860 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:55.685868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:55.685966 (+ 98us) server_negotiation.cc:407] Connection header received
0926 20:57:55.686180 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:55.686185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:55.686272 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:55.686384 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:55.687270 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.687958 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.689117 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:55.689580 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:55.689711 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:55.693082 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:55.693089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:55.693094 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:55.693136 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:55.696348 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:55.696977 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:55.696983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:55.696986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:55.697125 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:55.697599 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:55.697604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:55.697606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:55.697956 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:55.698098 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:55.698503 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 20:57:55.698714 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:57:55.699803 32162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } attempt: 2
I20260926 20:57:55.700289 32088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:57:55.700827 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.700314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53372)
0926 20:57:55.700550 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.700628 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 20:57:55.701488 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.700934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53388)
0926 20:57:55.701164 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.701303 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 20:57:55.701932 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.701471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53392)
0926 20:57:55.701711 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.701794 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260926 20:57:55.702171 32162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:33291: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
I20260926 20:57:55.702553 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.702079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53406)
0926 20:57:55.702267 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.702363 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 20:57:55.702867 32088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:33291: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
I20260926 20:57:55.754161 32088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } attempt: 1
I20260926 20:57:55.755609 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.754986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53416)
0926 20:57:55.755232 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.755365 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 20:57:55.756784 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.756330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53428)
0926 20:57:55.756522 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.756626 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 20:57:55.757076 32088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:33291: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
I20260926 20:57:55.793428 32162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } attempt: 3
I20260926 20:57:55.794644 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.794113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53434)
0926 20:57:55.794343 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.794454 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 20:57:55.795797 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.795331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53438)
0926 20:57:55.795568 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.795656 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260926 20:57:55.796000 32162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:33291: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
I20260926 20:57:55.825306 32088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } attempt: 2
I20260926 20:57:55.826522 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.825948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53448)
0926 20:57:55.826190 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.826314 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 20:57:55.827513 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.827153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53454)
0926 20:57:55.827316 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.827396 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:57:55.827723 32088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:33291: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
W20260926 20:57:55.927337 32170 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:57:55.927717 32170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:57:55.927812 32170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:57:55.927938 32170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 20:57:55.933945 32088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } attempt: 3
I20260926 20:57:55.935143 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.934590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53456)
0926 20:57:55.934829 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.934949 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260926 20:57:55.935962 32170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:57:55.936069 32170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:57:55.936158 32170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:57:55.936236 32170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:57:55.936326 32170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:57:55.936331 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.935840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53468)
0926 20:57:55.936076 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.936177 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260926 20:57:55.936592 32088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:33291: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
I20260926 20:57:55.939210 32162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } attempt: 4
I20260926 20:57:55.940369 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.939755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53470)
0926 20:57:55.940029 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.940149 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 20:57:55.941464 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:55.941038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53476)
0926 20:57:55.941230 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:55.941321 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 20:57:55.941681 32162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.249.188:33291: Network error: Client connection negotiation failed: client connection to 127.8.249.188:33291: connect: Connection refused (error 111)
I20260926 20:57:55.948325 32170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.249.148:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.249.190:36141,127.8.249.189:44175,127.8.249.188:33291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/krb5kdc/kudu_127.8.249.188.keytab
--principal=kudu/127.8.249.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.249.188:33291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.249.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 19:19:52 UTC on ccycloud.quasar-vsunte.root.comops.site
build id 4675017
ASAN enabled
I20260926 20:57:55.949824 32170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:57:55.951437 32170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:57:55.964973 32176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:55.965008 32178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:57:55.965230 32175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:57:55.966041 32170 server_base.cc:1034] running on GCE node
I20260926 20:57:55.967031 32170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:57:55.968542 32170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:57:55.969772 32170 hybrid_clock.cc:648] HybridClock initialized: now 1790456275969704 us; error 50 us; skew 500 ppm
I20260926 20:57:55.970145 32170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 20:57:55 dist-test-slave-2dg3 krb5kdc[32008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456275, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:57:55.973573 32170 init.cc:377] Logged in from keytab as kudu/127.8.249.188@KRBTEST.COM (short username kudu)
I20260926 20:57:55.975117 32170 webserver.cc:460] Webserver started at http://127.8.249.188:42741/ using document root <none> and password file <none>
I20260926 20:57:55.975898 32170 fs_manager.cc:359] Metadata directory not provided
I20260926 20:57:55.976028 32170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:57:55.976356 32170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:57:55.979367 32170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/data/instance:
uuid: "78dd66dbae36421ca07b7a50e7f2a278"
format_stamp: "Formatted at 2026-09-26 20:57:55 on dist-test-slave-2dg3"
server_key: "e014e22d81baa1ab03f21194f76d8cc9"
server_key_iv: "5cf273fe586ee0e766e23aeabfb03dee"
server_key_version: "encryptionkey@0"
I20260926 20:57:55.980134 32170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/wal/instance:
uuid: "78dd66dbae36421ca07b7a50e7f2a278"
format_stamp: "Formatted at 2026-09-26 20:57:55 on dist-test-slave-2dg3"
server_key: "e014e22d81baa1ab03f21194f76d8cc9"
server_key_iv: "5cf273fe586ee0e766e23aeabfb03dee"
server_key_version: "encryptionkey@0"
I20260926 20:57:55.985241 32170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 20:57:55.988667 32185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:57:55.989984 32170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:57:55.990159 32170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/data,/tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/wal
uuid: "78dd66dbae36421ca07b7a50e7f2a278"
format_stamp: "Formatted at 2026-09-26 20:57:55 on dist-test-slave-2dg3"
server_key: "e014e22d81baa1ab03f21194f76d8cc9"
server_key_iv: "5cf273fe586ee0e766e23aeabfb03dee"
server_key_version: "encryptionkey@0"
I20260926 20:57:55.990386 32170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:57:56.011137 32170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:57:56.015789 32170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:57:56.016079 32170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:57:56.035615 32170 rpc_server.cc:307] RPC server started. Bound to: 127.8.249.188:33291
I20260926 20:57:56.035635 32236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.249.188:33291 every 8 connection(s)
I20260926 20:57:56.037001 32170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/data/info.pb
I20260926 20:57:56.040725 32237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:57:56.043177 9190 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu as pid 32170
I20260926 20:57:56.043378 9190 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpGD2b0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790455916331309-9190-0/minicluster-data/master-2/wal/instance
I20260926 20:57:56.043848 9190 external_mini_cluster.cc:1428] Setting key ca3ec807ab908b8129d83bbedd47a6e3
I20260926 20:57:56.048385 32237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:57:56 dist-test-slave-2dg3 krb5kdc[32008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.188@KRBTEST.COM
Sep 26 20:57:56 dist-test-slave-2dg3 krb5kdc[32008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456275, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.188@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:56.068176 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.051559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.8.249.190:36141)
0926 20:57:56.051777 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.051784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.052225 (+ 441us) server_negotiation.cc:407] Connection header received
0926 20:57:56.052480 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.052484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.052551 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.052667 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.054115 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.055104 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.056941 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.057366 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.057460 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.061379 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:56.061386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:56.061392 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:56.061436 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:56.064907 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.065779 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.065785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.065789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.065924 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.066318 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.066324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.066327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.066597 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:56.066771 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.067364 (+ 593us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.067810 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:57:56.068907 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.045704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.8.249.188:33291)
0926 20:57:56.046519 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.046530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.046568 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:57:56.048239 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.048262 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.048700 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.049052 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.050910 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.052769 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.054475 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.055496 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.055630 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.060436 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:56.060446 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:56.060459 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:56.060489 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:56.064386 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.065339 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.065354 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.065356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.065518 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.065993 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.065997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.065999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.066550 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:56.066882 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.067254 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.067758 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":606,"thread_start_us":363,"threads_started":1}
I20260926 20:57:56.068925 32241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.051402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.190:36141 (local address 127.0.0.1:34436)
0926 20:57:56.052088 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:56.052128 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:56.052305 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:56.052727 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:56.052734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:56.053630 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:56.053957 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:56.053970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.055261 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.055265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:56.056780 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:56.056797 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.057612 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.057620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:56.057859 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.058508 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:56.058543 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:56.061114 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:56.065047 (+ 3933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:56.065061 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:56.065076 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:56.065625 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:56.066041 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:56.066046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:56.066050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:56.066204 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:56.066756 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:56.066764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:56.066987 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 20:57:56.067258 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 20:57:56.067932 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":168,"thread_start_us":281,"threads_started":1}
I20260926 20:57:56.071288 32237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:57:56 dist-test-slave-2dg3 krb5kdc[32008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456275, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.188@KRBTEST.COM for kudu/127.8.249.189@KRBTEST.COM
Sep 26 20:57:56 dist-test-slave-2dg3 krb5kdc[32008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.190@KRBTEST.COM
I20260926 20:57:56.085843 32241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.072111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.189:44175 (local address 127.0.0.1:43998)
0926 20:57:56.072354 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:56.072371 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:56.072465 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:56.072788 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:56.072793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:56.073327 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:56.073564 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:56.073577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.074941 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.074948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:56.075843 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:56.075855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.076577 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.076583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:56.076817 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.077474 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:56.077501 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:56.079790 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:56.083295 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:56.083301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:56.083306 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:56.083698 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:56.084112 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:56.084118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:56.084121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:56.084290 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:56.084841 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:56.084846 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:56.085006 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 20:57:56.085151 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 20:57:56.085468 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260926 20:57:56.085873 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.072187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43998 (local address 127.8.249.189:44175)
0926 20:57:56.072351 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.072355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.072441 (+ 86us) server_negotiation.cc:407] Connection header received
0926 20:57:56.072565 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.072570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.072645 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.072756 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.073791 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.074732 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.075996 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.076423 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.076562 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.080017 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:56.080021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:56.080025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:56.080060 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:56.083135 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.083836 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.083843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.083851 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.083990 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.084405 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.084411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.084414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.084714 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:56.084878 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.085250 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.085469 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:57:56.086607 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.072100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.8.249.190:36141)
0926 20:57:56.072437 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.072443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.072460 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:57:56.072568 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.072572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.072645 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.072747 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.074392 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.075151 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.076371 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.076784 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.076923 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.081117 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:56.081122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:56.081126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:56.081161 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:56.083940 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.084631 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.084638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.084641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.084767 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.085138 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.085144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.085148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.085419 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:56.085719 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.086006 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.086224 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:57:56.087512 32237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:57:56 dist-test-slave-2dg3 krb5kdc[32008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456275, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.188@KRBTEST.COM for kudu/127.8.249.188@KRBTEST.COM
Sep 26 20:57:56 dist-test-slave-2dg3 krb5kdc[32008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.249.189@KRBTEST.COM
I20260926 20:57:56.102033 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.088431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.8.249.188:33291)
0926 20:57:56.088587 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.088592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.088610 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:57:56.088767 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.088772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.088842 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.088946 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.089876 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.090669 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.092213 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.092601 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.092736 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.096058 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:56.096063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:56.096066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:56.096093 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:56.099051 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.099842 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.099849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.099854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.100009 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.100482 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.100486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.100489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.100795 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:56.100920 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.101361 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.101647 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":55}
I20260926 20:57:56.102070 32241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.088298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53484)
0926 20:57:56.088536 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:56.088556 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:56.088656 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:56.088988 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:56.088993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:56.089508 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:56.089717 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:56.089733 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.090855 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.090861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:56.092075 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:56.092089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.092767 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.092772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:56.092949 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.093608 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:56.093634 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:56.095868 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:56.099186 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:56.099192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:56.099197 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:56.099717 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:56.100132 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:56.100137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:56.100143 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:56.100315 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:56.100927 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:56.100935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:56.101102 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 20:57:56.101267 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 20:57:56.101795 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":195}
I20260926 20:57:56.104357 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.088425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.8.249.189:44175)
0926 20:57:56.088586 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.088592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.088610 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:57:56.088767 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.088772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.088856 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.088946 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.090408 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.091275 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.092729 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.093193 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.093326 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.098109 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:56.098114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:56.098117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:56.098149 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:56.101392 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.102343 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.102347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.102350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.102505 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.102943 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.102949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.102954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.103228 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:56.103494 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.103814 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.104042 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:57:56.108222 32237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278: Bootstrap starting.
I20260926 20:57:56.109450 9190 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:57:56.111701 32237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:56.112737 32237 log.cc:824] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:56.113826 32088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } attempt: 4
I20260926 20:57:56.111766 32063 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "089dc8cc023747bdbc16668bad02715c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34444
I20260926 20:57:56.116065 32237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278: No bootstrap required, opened a new log
Sep 26 20:57:56 dist-test-slave-2dg3 krb5kdc[32008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456274, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.190@KRBTEST.COM for kudu/127.8.249.188@KRBTEST.COM
I20260926 20:57:56.126248 32237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } }
I20260926 20:57:56.126726 32237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78dd66dbae36421ca07b7a50e7f2a278, State: Initialized, Role: FOLLOWER
I20260926 20:57:56.127303 32237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } }
I20260926 20:57:56.128278 32243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } } }
I20260926 20:57:56.128734 32243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:57:56.129102 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.114461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.8.249.188:33291)
0926 20:57:56.114662 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.114668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.114776 (+ 108us) server_negotiation.cc:407] Connection header received
0926 20:57:56.115011 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.115018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.115081 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.115182 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.115982 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.117125 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.118364 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.118951 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.119105 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.122269 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:56.122276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:56.122286 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:56.122323 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:56.126090 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.126774 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.126780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.126784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.126985 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.127393 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.127399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.127403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.127902 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:56.128088 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.128536 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.128791 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:57:56.129230 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.114417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53496)
0926 20:57:56.114669 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:56.114689 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:56.114883 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:56.115208 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:56.115213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:56.115695 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:56.115844 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:56.115852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.117268 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.117273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:56.118218 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:56.118229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.119112 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.119118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:56.119341 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.119951 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:56.119977 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:56.122044 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:56.126244 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:56.126249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:56.126254 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:56.126649 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:56.127104 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:56.127109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:56.127113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:56.127274 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:56.128056 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:56.128061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:56.128236 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 20:57:56.128396 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 20:57:56.128777 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260926 20:57:56.129698 32237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:56.135421 32088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c: Bootstrap starting.
I20260926 20:57:56.139652 32088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c: Neither blocks nor log segments found. Creating new log.
W20260926 20:57:56.139662 32254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:57:56.139768 32254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:57:56.140834 32088 log.cc:824] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:56.143693 32088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c: No bootstrap required, opened a new log
I20260926 20:57:56.137176 32170 master_runner.cc:186] Error getting master registration for 127.8.249.190:36141: OK, instance_id { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" instance_seqno: 1790456274970246 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:57:56.147859 32170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:57:56.149226 32088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } }
I20260926 20:57:56.149581 32088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089dc8cc023747bdbc16668bad02715c, State: Initialized, Role: FOLLOWER
I20260926 20:57:56.150147 32088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } }
I20260926 20:57:56.151196 32257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } } }
I20260926 20:57:56.151580 32257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:57:56.152397 32088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:56.152508 32063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 20:57:56.152706 32063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:56.156308 32063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } }
I20260926 20:57:56.159413 32063 leader_election.cc:288] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [CANDIDATE]: Term 1 election: Requested vote from peers 85173e5e4ca54fd9bf4cdd5896e47660 (127.8.249.189:44175), 78dd66dbae36421ca07b7a50e7f2a278 (127.8.249.188:33291)
I20260926 20:57:56.159934 32212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "089dc8cc023747bdbc16668bad02715c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "78dd66dbae36421ca07b7a50e7f2a278"
I20260926 20:57:56.160392 32212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:56.163623 32212 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 089dc8cc023747bdbc16668bad02715c in term 1.
I20260926 20:57:56.164529 32040 leader_election.cc:302] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 089dc8cc023747bdbc16668bad02715c, 78dd66dbae36421ca07b7a50e7f2a278; no voters:
I20260926 20:57:56.165138 32257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:57:56.179522 32257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [term 1 LEADER]: Becoming Leader. State: Replica: 089dc8cc023747bdbc16668bad02715c, State: Running, Role: LEADER
I20260926 20:57:56.180387 32257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } }
I20260926 20:57:56.191177 32269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 089dc8cc023747bdbc16668bad02715c. Latest consensus state: current_term: 1 leader_uuid: "089dc8cc023747bdbc16668bad02715c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } } }
I20260926 20:57:56.191804 32269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:56.198441 32274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260926 20:57:56.209713 32042 tablet.cc:2369] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:57:56.218466 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.164162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44016 (local address 127.8.249.189:44175)
0926 20:57:56.164349 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.164356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.176254 (+ 11898us) server_negotiation.cc:407] Connection header received
0926 20:57:56.176775 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.176783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.176885 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.176983 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.181900 (+ 4917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.183138 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.186863 (+ 3725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.187650 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.187794 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.205034 (+ 17240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:56.205043 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:56.205049 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:56.205106 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:56.209469 (+ 4363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.215734 (+ 6265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.215741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.215744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.215977 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.216493 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.216501 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.216504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.216915 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:56.217172 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.217803 (+ 631us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.218112 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 20:57:56.220665 32274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:57:56.223793 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.194224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.8.249.188:33291)
0926 20:57:56.194823 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.194830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.201559 (+ 6729us) server_negotiation.cc:407] Connection header received
0926 20:57:56.202105 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.202111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.202210 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.202285 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.204619 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.205701 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.207642 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.208355 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.208468 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.213279 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:56.213288 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:56.213293 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:56.213345 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:56.217592 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.219357 (+ 1765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.219365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.219369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.219589 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.221111 (+ 1522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.221118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.221122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.221555 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:56.221839 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.223042 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.223341 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406}
I20260926 20:57:56.229943 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.163772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.8.249.190:36141)
0926 20:57:56.163955 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.163962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.193962 (+ 30000us) server_negotiation.cc:407] Connection header received
0926 20:57:56.194279 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.194285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.194406 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.194491 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.201588 (+ 7097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.202753 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.206960 (+ 4207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.207714 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.207821 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.218998 (+ 11177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:56.219006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:56.219012 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:56.219058 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:56.224392 (+ 5334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.225648 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.225654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.225658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.225846 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.227950 (+ 2104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.227955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.227959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.228369 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:56.228647 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.229241 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.229515 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:57:56.240866 32162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } attempt: 5
I20260926 20:57:56.250514 32212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278 [term 1 FOLLOWER]: Refusing update from remote peer 089dc8cc023747bdbc16668bad02715c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:57:56.251623 32269 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [LEADER]: Connected to new peer: Peer: permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 26 20:57:56 dist-test-slave-2dg3 krb5kdc[32008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790456275, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.249.189@KRBTEST.COM for kudu/127.8.249.188@KRBTEST.COM
W20260926 20:57:56.268327 32191 tablet.cc:2369] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:57:56.248827 32021 master_runner.cc:186] Error getting master registration for 127.8.249.189:44175: OK, instance_id { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" instance_seqno: 1790456275513135 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:57:56.272799 32021 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:57:56.273208 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.241524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.249.188:33291 (local address 127.0.0.1:53516)
0926 20:57:56.241809 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 20:57:56.241832 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 20:57:56.241954 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:57:56.242513 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:57:56.242518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:57:56.243090 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:57:56.243306 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:56.243321 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.244769 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.244776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:56.246224 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:57:56.246243 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.252949 (+ 6706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.252958 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:57:56.253281 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.254383 (+ 1102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:57:56.254515 (+ 132us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:57:56.262970 (+ 8455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:57:56.268160 (+ 5190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:56.268170 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:56.268174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:56.269214 (+ 1040us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:56.269965 (+ 751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:57:56.269971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:57:56.269976 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:57:56.270307 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:57:56.271260 (+ 953us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:57:56.271266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:57:56.271578 (+ 312us) client_negotiation.cc:769] Sending connection context
0926 20:57:56.271856 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 20:57:56.272465 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260926 20:57:56.273859 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.241689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.8.249.188:33291)
0926 20:57:56.241876 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.241883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.241906 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:57:56.242098 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.242104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.242184 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.242291 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.243497 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.244560 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.246469 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.247237 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.247377 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.263390 (+ 16013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:57:56.263400 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:57:56.263408 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:57:56.263462 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:57:56.267922 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.269420 (+ 1498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.269430 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.269434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.269762 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:57:56.270503 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:57:56.270508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:57:56.270511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:57:56.271041 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:57:56.271200 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.272082 (+ 882us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.272618 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:57:56.280334 32162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660: Bootstrap starting.
I20260926 20:57:56.287712 32162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660: Neither blocks nor log segments found. Creating new log.
I20260926 20:57:56.288122 32243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 089dc8cc023747bdbc16668bad02715c. Latest consensus state: current_term: 1 leader_uuid: "089dc8cc023747bdbc16668bad02715c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } } }
I20260926 20:57:56.288513 32243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:57:56.290547 32162 log.cc:824] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660: Log is configured to *not* fsync() on all Append() calls
I20260926 20:57:56.298810 32257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "089dc8cc023747bdbc16668bad02715c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } } }
I20260926 20:57:56.299239 32257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:56.300218 32243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "089dc8cc023747bdbc16668bad02715c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } } }
I20260926 20:57:56.300587 32243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:57:56.301440 32281 mvcc.cc:205] Tried to move back new op lower bound from 7333708907486695424 to 7333708907242901504. Current Snapshot: MvccSnapshot[applied={T|T < 7333708907486695424}]
I20260926 20:57:56.308290 32274 catalog_manager.cc:1347] Generated new cluster ID: 9cf846f670c44e15abc1959a37c1b6d2
I20260926 20:57:56.308429 32274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:57:56.311730 32162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660: No bootstrap required, opened a new log
I20260926 20:57:56.323010 32162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } }
I20260926 20:57:56.323654 32162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85173e5e4ca54fd9bf4cdd5896e47660, State: Initialized, Role: FOLLOWER
I20260926 20:57:56.324272 32162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } }
I20260926 20:57:56.331415 32286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } } }
I20260926 20:57:56.333928 32286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:57:56.333774 32162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:57:56.334213 32136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:57:56.338922 32137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "089dc8cc023747bdbc16668bad02715c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "85173e5e4ca54fd9bf4cdd5896e47660"
I20260926 20:57:56.340013 32136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660 [term 1 FOLLOWER]: Refusing update from remote peer 089dc8cc023747bdbc16668bad02715c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:57:56.349467 32137 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 089dc8cc023747bdbc16668bad02715c in term 1.
I20260926 20:57:56.350564 32269 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:57:56.393761 32274 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:57:56.396437 32274 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:57:56.407473 32257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "089dc8cc023747bdbc16668bad02715c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } } }
I20260926 20:57:56.407809 32257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c [sys.catalog]: This master's current role is: LEADER
I20260926 20:57:56.409924 32293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "089dc8cc023747bdbc16668bad02715c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } } }
I20260926 20:57:56.410486 32293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:57:56.410059 32286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 089dc8cc023747bdbc16668bad02715c. Latest consensus state: current_term: 1 leader_uuid: "089dc8cc023747bdbc16668bad02715c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "089dc8cc023747bdbc16668bad02715c" member_type: VOTER last_known_addr { host: "127.8.249.190" port: 36141 } } peers { permanent_uuid: "85173e5e4ca54fd9bf4cdd5896e47660" member_type: VOTER last_known_addr { host: "127.8.249.189" port: 44175 } } peers { permanent_uuid: "78dd66dbae36421ca07b7a50e7f2a278" member_type: VOTER last_known_addr { host: "127.8.249.188" port: 33291 } } }
I20260926 20:57:56.410856 32286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:57:56.433795 32274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 089dc8cc023747bdbc16668bad02715c: Generated new TSK 0
I20260926 20:57:56.435072 32274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:57:56.474344 32303 catalog_manager.cc:1259] Loaded cluster ID: 9cf846f670c44e15abc1959a37c1b6d2
I20260926 20:57:56.474478 32303 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660: loading cluster ID for follower catalog manager: success
I20260926 20:57:56.478639 32303 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660: acquiring CA information for follower catalog manager: success
I20260926 20:57:56.481395 32303 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85173e5e4ca54fd9bf4cdd5896e47660: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 20:57:56.548367 32167 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:56.543809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.8.249.189:44175)
0926 20:57:56.544026 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.544035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.545736 (+ 1701us) server_negotiation.cc:407] Connection header received
0926 20:57:56.546735 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.546741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.546906 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.547008 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.548127 (+ 1119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44024: BlockingRecv error: recv got EOF from 127.0.0.1:44024 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 20:57:56.552443 32165 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:56.543477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34470 (local address 127.8.249.190:36141)
0926 20:57:56.543676 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.543682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.550237 (+ 6555us) server_negotiation.cc:407] Connection header received
0926 20:57:56.550946 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.550953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.551039 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.551118 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.552147 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34470: BlockingRecv error: recv got EOF from 127.0.0.1:34470 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 20:57:56.556028 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:56.553607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.8.249.188:33291)
0926 20:57:56.553817 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.553822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.554161 (+ 339us) server_negotiation.cc:407] Connection header received
0926 20:57:56.554456 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.554463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.554555 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.554630 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.555794 (+ 1164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53526: BlockingRecv error: recv got EOF from 127.0.0.1:53526 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 20:57:56.623716 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.597244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.8.249.189:44175)
0926 20:57:56.597491 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.597500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.615456 (+ 17956us) server_negotiation.cc:407] Connection header received
0926 20:57:56.615737 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.615743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.615972 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.616152 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:56.616910 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.618097 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.620269 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.620966 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.621096 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.621791 (+ 695us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:56.622544 (+ 753us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:56.622694 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.623216 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.623369 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 20:57:56.677146 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:56.667865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.8.249.188:33291)
0926 20:57:56.668111 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.668121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.675013 (+ 6892us) server_negotiation.cc:407] Connection header received
0926 20:57:56.675557 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.675563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.675716 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.675799 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.676886 (+ 1087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53534: BlockingRecv error: recv got EOF from 127.0.0.1:53534 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 20:57:56.729398 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.710874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44042 (local address 127.8.249.189:44175)
0926 20:57:56.711091 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.711097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.722066 (+ 10969us) server_negotiation.cc:407] Connection header received
0926 20:57:56.722699 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.722706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.722848 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.722922 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:56.723677 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.724885 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.726977 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.727678 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.727775 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.728440 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:56.728637 (+ 197us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:56.728704 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.728947 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.729044 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 20:57:56.777079 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:56.767038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.8.249.188:33291)
0926 20:57:56.767230 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.767235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.775244 (+ 8009us) server_negotiation.cc:407] Connection header received
0926 20:57:56.775550 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.775558 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.775653 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.775733 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.776804 (+ 1071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53544: BlockingRecv error: recv got EOF from 127.0.0.1:53544 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:57:56.838357 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.831185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.8.249.189:44175)
0926 20:57:56.831391 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.831396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.831424 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:57:56.831770 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.831777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.831873 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.831986 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:56.832463 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.833746 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.835992 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.836715 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.836858 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.837356 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:56.837512 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:56.837617 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.837895 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.838006 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 20:57:56.848006 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:56.845761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.8.249.188:33291)
0926 20:57:56.845946 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.845952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.846516 (+ 564us) server_negotiation.cc:407] Connection header received
0926 20:57:56.846681 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.846689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.846811 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.846917 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.847765 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53552: BlockingRecv error: recv got EOF from 127.0.0.1:53552 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:57:56.862042 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.855138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.8.249.189:44175)
0926 20:57:56.855366 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.855372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.855857 (+ 485us) server_negotiation.cc:407] Connection header received
0926 20:57:56.856036 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.856042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.856126 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.856229 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:56.856735 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.857920 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.859904 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.860619 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.860752 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.861255 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:56.861408 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:56.861524 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.861680 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.861748 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260926 20:57:56.870934 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:56.869022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.8.249.188:33291)
0926 20:57:56.869312 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.869318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.869442 (+ 124us) server_negotiation.cc:407] Connection header received
0926 20:57:56.869618 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.869622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.869696 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.869778 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.870625 (+ 847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53558: BlockingRecv error: recv got EOF from 127.0.0.1:53558 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
I20260926 20:57:56.888365 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.882078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.8.249.189:44175)
0926 20:57:56.882307 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.882314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.882602 (+ 288us) server_negotiation.cc:407] Connection header received
0926 20:57:56.882924 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.882929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.883018 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.883119 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:56.883597 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.884399 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.886299 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.886911 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.887023 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.887499 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:56.887617 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:56.887694 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.887907 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.887986 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260926 20:57:56.897377 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:56.895199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.8.249.188:33291)
0926 20:57:56.895430 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.895435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.895781 (+ 346us) server_negotiation.cc:407] Connection header received
0926 20:57:56.895961 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.895965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.896041 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.896132 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.897052 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53562: BlockingRecv error: recv got EOF from 127.0.0.1:53562 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 20:57:56.923475 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.917402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.8.249.189:44175)
0926 20:57:56.917637 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.917646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.918037 (+ 391us) server_negotiation.cc:407] Connection header received
0926 20:57:56.918211 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.918216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.918304 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.918420 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:56.918826 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.920004 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.921316 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.922010 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.922137 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.922565 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:56.922694 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:56.922811 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.923059 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.923150 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 20:57:56.932150 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:56.930056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.8.249.188:33291)
0926 20:57:56.930286 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.930294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.930638 (+ 344us) server_negotiation.cc:407] Connection header received
0926 20:57:56.930775 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.930781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.930921 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.931094 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.931905 (+ 811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53566: BlockingRecv error: recv got EOF from 127.0.0.1:53566 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:57:56.974769 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:56.968182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.8.249.189:44175)
0926 20:57:56.968430 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.968436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.968864 (+ 428us) server_negotiation.cc:407] Connection header received
0926 20:57:56.969129 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.969134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.969226 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.969323 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:56.969845 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.970764 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.972869 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:56.973332 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:56.973443 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:56.973985 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:56.974105 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:56.974180 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:56.974354 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 20:57:56.974425 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260926 20:57:56.986946 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:56.984698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53568 (local address 127.8.249.188:33291)
0926 20:57:56.984882 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:56.984890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:56.985369 (+ 479us) server_negotiation.cc:407] Connection header received
0926 20:57:56.985560 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:56.985566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:56.985677 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:56.985813 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:56.986672 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53568: BlockingRecv error: recv got EOF from 127.0.0.1:53568 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:57:57.059036 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:57.054249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.8.249.189:44175)
0926 20:57:57.054417 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:57.054422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:57.054888 (+ 466us) server_negotiation.cc:407] Connection header received
0926 20:57:57.055061 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:57.055065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:57.055136 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:57.055239 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:57.055625 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:57.056390 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:57.057616 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:57.058016 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:57.058106 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:57.058448 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:57.058542 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:57.058605 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:57.058721 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 20:57:57.058785 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 20:57:57.066701 32240 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:57:57.064849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.8.249.188:33291)
0926 20:57:57.065078 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:57.065086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:57.065331 (+ 245us) server_negotiation.cc:407] Connection header received
0926 20:57:57.065543 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:57.065555 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:57.065637 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:57.065739 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:57:57.066543 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53580: BlockingRecv error: recv got EOF from 127.0.0.1:53580 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 20:57:57.141716 32254 catalog_manager.cc:1259] Loaded cluster ID: 9cf846f670c44e15abc1959a37c1b6d2
I20260926 20:57:57.141827 32254 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278: loading cluster ID for follower catalog manager: success
I20260926 20:57:57.144408 32254 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278: acquiring CA information for follower catalog manager: success
I20260926 20:57:57.146167 32254 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 78dd66dbae36421ca07b7a50e7f2a278: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:57:57.203833 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:57.198634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.8.249.189:44175)
0926 20:57:57.198904 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:57.198908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:57.199285 (+ 377us) server_negotiation.cc:407] Connection header received
0926 20:57:57.199439 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:57.199443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:57.199519 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:57.199616 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:57.200061 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:57.200786 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:57.202226 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:57.202663 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:57.202763 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:57.203201 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:57.203328 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:57.203401 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:57.203540 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 20:57:57.203611 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 20:57:57.220520 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:57.210521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53594 (local address 127.8.249.188:33291)
0926 20:57:57.210776 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:57.210806 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:57.214355 (+ 3549us) server_negotiation.cc:407] Connection header received
0926 20:57:57.214604 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:57.214609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:57.214927 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:57.215065 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:57.215600 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:57.216757 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:57.218128 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:57.218562 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:57.218723 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:57.219320 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:57.219853 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:57.220025 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:57.220106 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:57:57.220213 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:57:57.235481 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:57.226985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.8.249.189:44175)
0926 20:57:57.227246 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:57.227255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:57.228584 (+ 1329us) server_negotiation.cc:407] Connection header received
0926 20:57:57.228812 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:57.228818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:57.228903 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:57.229017 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:57.229506 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:57.230378 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:57.233040 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:57.233850 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:57.234001 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:57.234580 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:57.234697 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:57.234803 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:57.235007 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:57:57.235098 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:57:57.238694 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:57.226938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.8.249.190:36141)
0926 20:57:57.228193 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:57.228201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:57.230710 (+ 2509us) server_negotiation.cc:407] Connection header received
0926 20:57:57.231172 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:57.231191 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:57.231448 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:57.231712 (+ 264us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:57.232541 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:57.233522 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:57.235714 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:57.236545 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:57.236647 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:57.237442 (+ 795us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:57.237957 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:57.238071 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:57.238206 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 20:57:57.238287 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":1011,"threads_started":1}
I20260926 20:57:57.239257 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:57:57.231078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.8.249.188:33291)
0926 20:57:57.231254 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:57:57.231258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:57:57.232551 (+ 1293us) server_negotiation.cc:407] Connection header received
0926 20:57:57.232892 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:57:57.232898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:57:57.233005 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:57:57.233105 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:57:57.233727 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:57.234972 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:57.237095 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:57:57.237779 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:57:57.237930 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:57:57.238442 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:57:57.238577 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:57:57.238690 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:57:57.238843 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 20:57:57.238937 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:57:57.248756 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 32021
I20260926 20:57:57.249868 32083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:57.365026 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 32021
I20260926 20:57:57.385975 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 32095
I20260926 20:57:57.387200 32157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:57.493889 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 32095
I20260926 20:57:57.514227 9190 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu pid 32170
I20260926 20:57:57.515439 32232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 20:57:57.622241 9190 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpGD2b0/build/asan/bin/kudu with pid 32170
2026-09-26T20:57:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4483 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8938 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (361258 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST